Commit Graph

Select branches

Hide Pull Requests

master

#1

1.89

Mono Color

• 4c8b74bd8f
  Merge pull request #1 from simon987/dependabot/pip/pydantic-1.10.13 master simon987 2024-04-25 08:45:06 -04:00
• d82e1bccee
  Bump pydantic from 1.10.11 to 1.10.13 dependabot[bot] 2024-04-25 05:06:23 +00:00
• b1a1da3bac Add option to use nltk word_tokenize 1.89 simon987 2023-09-09 11:11:44 -04:00
• a047366926 pin pydantic version simon987 2023-07-13 08:27:48 -04:00
• 24230cdc1e Update PgConn simon987 2023-05-26 14:09:45 -04:00
• 3bd9f03996 Fix PersistentState constructor simon987 2023-04-10 17:01:42 -04:00
• e267bbf1c8 Set json indent=2 for pydantic rows simon987 2023-02-25 16:07:41 -05:00
• 42e33b72b2 Fix JSON encoding for Enums simon987 2023-02-25 15:51:28 -05:00
• 5275c332cc Add drop table simon987 2023-02-25 15:38:40 -05:00
• a7b1a6e1ec Fix tests, add pydantic row support for PersistentState simon987 2023-02-25 15:20:17 -05:00
• 826312115c Fix deserialization in PersistentState again simon987 2022-05-07 09:41:10 -04:00
• 372abb0076 Fix deserialization in PersistentState simon987 2022-05-07 09:34:50 -04:00
• 78c04ef6f3 Add option to override Table factory in PersistentState simon987 2022-05-05 15:02:48 -04:00
• a51ad2cbb4 Cleanup simon987 2022-05-03 10:59:25 -04:00
• 4befc3973d Add strip_dashes option in preprocess() simon987 2022-02-26 19:31:22 -05:00
• c9fac7151a Split punctuation into punctuation and special_punctuation simon987 2022-02-23 11:01:17 -05:00
• 084acbe184 Set max_window_size=2147483648 for zstd simon987 2022-01-29 10:44:38 -05:00
• d578be3218 Increase timeout simon987 2022-01-29 10:38:23 -05:00
• cd5a1ac50c Remove clean_multicore function simon987 2022-01-28 20:11:26 -05:00
• 62e74ed292 Make sure that _id field is present in redis MQ simon987 2022-01-27 11:06:52 -05:00
• 428c82bcfd Update retry codes simon987 2021-12-09 19:46:12 -05:00
• 4b3583358b Update retry codes simon987 2021-12-09 19:44:21 -05:00
• 90d434ec73 Add more single quotes simon987 2021-11-16 15:32:24 -05:00
• 55fd4a66d2 Fix strip_quotes simon987 2021-11-16 11:48:23 -05:00
• 3677815d57 Add more quotes in strip_quotes simon987 2021-11-16 11:39:10 -05:00
• 1ce795a759 ... simon987 2021-11-16 11:36:17 -05:00
• e1537297d7 normalize dashes in preprocess simon987 2021-11-16 11:34:48 -05:00
• 8d8f9e8751 Fix typo, add stateless stream processor simon987 2021-11-04 13:23:23 -04:00
• 18ba0024ea Null check on logger simon987 2021-11-03 16:47:21 -04:00
• 408735a926 Fix git clone simon987 2021-11-02 14:38:12 -04:00
• 2f6c2822b6 Add functions to handle routing keys simon987 2021-10-22 10:36:05 -04:00
• d85ad919b3 Add redis MQ, update influxdb monitoring simon987 2021-10-21 20:19:09 -04:00
• ed9d148411 Fix tests simon987 2021-10-21 19:52:42 -04:00
• 5e00ddccdb Add test file simon987 2021-10-07 15:40:23 -04:00
• 7ecd55a1c6 Add cookiejar_filter_name simon987 2021-10-03 11:02:39 -04:00
• b746a91281 Fix default value for retry codes simon987 2021-10-03 10:43:19 -04:00
• 333083e8b9 Add 520 in default retry codes simon987 2021-10-03 10:33:49 -04:00
• c295b5d30b Add Web.post simon987 2021-09-28 15:08:51 -04:00
• da0e117550 Fix fake-useragent (for real time time?) simon987 2021-09-25 16:03:45 -04:00
• c3fef7e7f8 Fix fake-useragent simon987 2021-09-25 15:53:52 -04:00
• 9bd1f4b799 unbreak statefulstreamworker simon987 2021-09-23 19:14:11 -04:00
• c560cc2010 tweak StatefulStreamWorker interface simon987 2021-09-19 14:19:17 -04:00
• f4a5e6cf53 queue_iter fix simon987 2021-09-19 12:44:49 -04:00
• 71cd00c063 Add StatfulStreamProcessor simon987 2021-09-19 12:39:57 -04:00
• 7349c9a5f1 Quick optimisation simon987 2021-09-19 10:57:07 -04:00
• d19442b00e Update preprocess: now returns generator objects simon987 2021-09-19 09:35:35 -04:00
• 4711cd1b66 Add trigrams simon987 2021-09-10 17:35:19 -04:00
• 7e0ffafb8c Update fix_single_quotes simon987 2021-08-28 20:48:00 -04:00
• 60273fb6bd Use imap instead of map simon987 2021-08-28 20:09:00 -04:00
• 67c09cc10c Add remove_numbers simon987 2021-08-28 20:06:53 -04:00
• a7bf5b2d15 Fix clean html (again!) simon987 2021-08-28 19:59:04 -04:00
• 31b35e3a32 Fix clean html (again) simon987 2021-08-28 19:44:10 -04:00
• 4cff343370 version bump simon987 2021-08-28 19:34:09 -04:00
• 4d6c8018df Fix clean_html simon987 2021-08-28 19:33:11 -04:00
• db3e191983 Add plot_confusion_matrix simon987 2021-06-29 14:13:07 -04:00
• 33e9734991 Add plot_freq_bar simon987 2021-06-23 19:37:22 -04:00
• 3238f92e4d Revert "get_soup() decode utf8" simon987 2021-05-14 10:09:07 -04:00
• f8e93354a4 get_soup() decode utf8 simon987 2021-05-14 09:59:59 -04:00
• 75bf2c2d85 Rename test.clean to text.preprocess, add QS util func, more debug logging simon987 2021-04-25 12:10:03 -04:00
• 9002ae7506 Add debug info simon987 2021-04-24 10:21:54 -04:00
• 88f3124f85 add bigram option for clean function simon987 2021-04-21 21:34:49 -04:00
• 8edad0255b add retries arg in get_web() simon987 2021-04-21 19:50:59 -04:00
• 32119535ae improve text cleaning simon987 2021-04-18 21:27:12 -04:00
• 2ffaa4a5b3 improve text cleaning simon987 2021-04-18 21:10:07 -04:00
• 067a20f7a8 improve text cleaning simon987 2021-04-18 20:32:34 -04:00
• 00323ea576 improve text cleaning simon987 2021-04-18 18:50:39 -04:00
• 45b5803c40 improve text cleaning simon987 2021-04-18 15:40:30 -04:00
• 18cd59fc4a ignore log in text simon987 2021-04-18 12:20:22 -04:00
• d895ac837e ignore log in text simon987 2021-04-18 12:18:27 -04:00
• ae59522b27 Add customstderr simon987 2021-04-18 12:17:00 -04:00
• 765f6f59b7 Add text cleaning function simon987 2021-04-18 12:12:31 -04:00
• 30902c8235 Add sep option in volatile state simon987 2021-04-16 19:10:45 -04:00
• 53a262a138 Add retry_sleep to retry simon987 2021-04-06 21:24:23 -04:00
• 6e1aa53455 Add retry_sleep to retry simon987 2021-04-06 21:23:28 -04:00
• 53ac0c37e8 Fix redis_publish simon987 2021-04-06 21:08:43 -04:00
• 8378ed6526 Add session arg to get_web simon987 2021-04-06 20:32:34 -04:00
• c79b3bfafd Add get_soup() simon987 2021-04-05 19:17:27 -04:00
• 5ee1629c79 oops simon987 2021-03-28 09:53:16 -04:00
• 00f5aef721 Update redis_publish to add subproject simon987 2021-03-28 09:42:52 -04:00
• 021da84433 add redis_publish simon987 2021-03-25 18:25:46 -04:00
• 53a03baaa4 Add useragent option in Web simon987 2021-03-07 11:02:26 -05:00
• 66d37e0be2 Add way to manually flush @buffered simon987 2021-02-28 12:15:44 -05:00
• 43cb6c4a7b web retry_codes fix simon987 2021-02-27 12:07:48 -05:00
• 4278b0f89e web retry_codes fix simon987 2021-02-27 08:58:21 -05:00
• 9738819428 web logging fix simon987 2021-02-25 22:00:44 -05:00
• ce2e5b2af6 load redis from env if not specified simon987 2021-02-25 21:35:18 -05:00
• 9cadce62ac add Web helper & logger simon987 2021-02-25 21:26:27 -05:00
• 7d330a0f9f Add pgsql wrapper & delitem for persistent state simon987 2021-02-06 15:41:18 -05:00
• a2cfab55bc msgpack for queue simon987 2021-01-20 20:30:57 -05:00
• c4fca1b754 Add volatile queue simon987 2021-01-20 20:08:35 -05:00
• d615ebdbd9 add custum SQL in persistant state simon987 2021-01-17 10:05:39 -05:00
• f914759b71 Fix @retry simon987 2021-01-10 22:46:20 -05:00
• 58d150279f Handle bool values in state simon987 2021-01-10 21:26:34 -05:00
• b2efaa99a4 Handle null values in state simon987 2021-01-10 20:56:46 -05:00
• b845d96295 Fix update in persistentstate simon987 2021-01-09 19:57:31 -05:00
• a1156599ce Fix tests simon987 2021-01-09 19:31:55 -05:00
• e3f3ea87eb Enable int IDs in PersistentState simon987 2021-01-09 19:29:25 -05:00
• 89b21884b7 Use hash in volatile state simon987 2020-12-20 20:15:17 -05:00
• 6832b9edd6 remove extraneous ttl arg simon987 2020-12-20 19:55:52 -05:00
• 30c9494daa add download_file, bool volatile state simon987 2020-12-20 19:53:38 -05:00