15 Commity (main)

Autor	SHA1	Wiadomość	Data
Štěpán Škorpil	87f5ddc8b1	If nodeinfo refresh is failed, software name and version is unset	2022-12-27 22:40:15 +01:00
Štěpán Škorpil	425abd5af0	Added domain validation	2022-12-10 15:11:56 +01:00
Štěpán Škorpil	1cbd5df5ae	Added config for seed domain timeout	2022-11-29 14:11:27 +01:00
Štěpán Škorpil	6a11d19781	Added crawling depth limit	2022-11-24 00:39:17 +01:00
Štěpán Škorpil	4064db521f	Added promise batching	2022-11-23 23:19:28 +01:00
Štěpán Škorpil	59f49ce29d	Changed tag from noindex to nobot	2022-11-22 18:39:05 +01:00
Štěpán Škorpil	c0b10f2e7a	It respects noindex option in mastodon	2022-11-22 17:35:34 +01:00
Štěpán Škorpil	4113d78a17	It respects noindex option in mastodon	2022-11-22 17:01:25 +01:00
Štěpán Škorpil	704c7c066e	Added robots.txt handling	2022-11-22 16:37:11 +01:00
Štěpán Škorpil	29acce3906	Properly applied code quality tools	2022-09-18 13:32:25 +02:00
Štěpán Škorpil	f6f9d758a1	Replaced postgresql by elastic search	2022-09-14 21:16:04 +02:00
Štěpán Škorpil	3ed37c04ce	Not existing feeds are deleted from databse	2022-04-17 14:03:53 +02:00
Štěpán Škorpil	b5df2b986a	Added addresses to fulltext	2022-01-01 12:21:43 +01:00
Štěpán Škorpil	d2ab8bfb2f	Add display name to fulltext field, fixed waiting when all feeds are processed	2021-12-28 00:20:19 +01:00
Štěpán Škorpil	74e7dca8bd	Initial crawler commit	2021-12-23 15:14:06 +01:00