wikitolearn/wikitolearn-content-migration/README.md

0001 # WikiToLearn Content Migration
0002
0003 ## bin/ content
0004 * `wtlm-cache-export.sh` export caches (parsoid and math-rendering example)
0005 * `wtlm-clean-destination.sh` deletes only destination databases
0006 * `wtlm-clean.sh` delete verything
0007 * `wtlm-down.sh` shutdown the system
0008 * `wtlm-output-export.sh` export destination mw pages collections
0009 * `wtlm-run.sh` run the pipeline
0010 * `wtlm-setup.sh` setup the system
0011 * `wtlm-ship.sh` run a little webui to test the data
0012 * `wtlm-update-production-v1-dump.sh` download the data from a backup NAS
0013
0014 ## pipeline scripts
0015 * `005-setup-source`
0016  * `005-wtl1-logging.py`
0017  * `010-wtl1-unified-revision.py`
0018  * `015-wtl1-superlog.py`
0019 * `010-setup-aux-db`
0020  * `005-create-index.py`
0021 * `015-import-pages-log`
0022  * `005-create-pages.py`
0023  * `010-create-pages-with-resused-ids.py`
0024  * `015-set-validity-range.py`
0025  * `020-superlog-load-base.py`
0026  * `025-delete-pages-with-wrong-ns.py`
0027  * `030-superlog-delete-with-last-operation-delete-before-day-zero.py`
0028  * `035-set-CourseRoot.py`
0029 * `020-fix-data`
0030  * `005-superlog-fix-params.py`
0031  * `010-superlog-sort-by-timestamp.py`
0032  * `015-superlog-fix-delete-actions.py`
0033  * `020-superlog-fix-titles-with-move-events.py`
0034  * `025-superlog-remove-items-before-day-zero.py`
0035  * `030-superlog-load-text.py`
0036  * `035-superlog-add-text-to-events.py`
0037  * `040-superlog-set-timestamp-next.py`
0038  * `045-delete-page-with-last-operation-a-delete.py`
0039 * `025-parse-struct-pages`
0040  * `005-CourseRoot.py`
0041  * `010-CourseMetadata.py`
0042  * `015-CourseLevelTwo.py`
0043  * `020-CourseLevelThree.py`
0044  * `025-delete-page-not-in-use.py`
0045 * `030-parse-content`
0046  * `005-parse-content.py`
0047  * `010-delete-redirects.py`
0048 * `035-fix-parsoid-cache`
0049  * `005-fix-parsoid-cache.py`
0050 * `040-fix-superlog`
0051  * `005-create-the-id-field-id-in-subpage-log.py`
0052  * `010-insert-page-creation.py`
0053  * `015-fix-all-used-titles.py`
0054  * `020-all-fix-users-ids.py`
0055 * `045-content-parsing`
0056  * `005-text-extract-objects.py`
0057  * `010-resolve-links.py`
0058  * `015-resolve-math.py`
0059 * `050-math-resolve`
0060  * `005-texvcjs.js`
0061  * `010-render.py`
0062  * `015-lookup-pages.py`
0063 * `055-file-resolve`
0064  * `005-load-files-revisions.py`
0065  * `010-download.py`
0066 * `060-mw-test-online`
0067  * `005-check-head-online-structure.py`
0068  * `010-check-head-online-wikitext.py`
0069  * `015-missing-pages.py`
0070 * `065-import-users-from-mw`
0071  * `005-import-user.py`
0072  * `010-test-users-with-B-passwords.py`
0073  * `015-prepare-keycloak-object.py`
0074 * `070-export-users-to-keycloak`
0075  * `005-keycloak-setup.py`
0076  * `010-upload-in-keycloak.py`
0077  * `015-anon-keycloak.py`
0078  * `020-keycloak-users-id-association.py`
0079 * `075-create-wtl-pages`
0080  * `005-create-wtl-pages.py`
0081  * `010-fix-titles-CourseLevelThree-and-CourseLevelTwo.py`
0082  * `015-fix-titles-CourseRoot.py`
0083  * `020-lookup-owner-CourseRoot.py`
0084  * `025-remove-duplicates.py`
0085 * `080-add-revisions`
0086  * `005-create-additional-revisions.py`
0087 * `085-eve-export`
0088  * `005-prepare-to-eve.py`
0089  * `010-pages.py`
0090  * `015-chapters-courses.py`
0091  * `020-coursessecurity.py`
0092 * `090-eve-fix`
0093  * `005-pages.py`
0094  * `010-chapters.py`
0095  * `015-courses.py`
0096  * `020-coursessecurity.py`