Arches 7.0.0 release notes
------------------------


### Major enhancements
- Adds frontend compiler ( webpack ), PR #8090
- Removes Arches Collector and associated references
- Upgrade ElasticSearch from 7.11.0 to v8.3.1, PR #8668
- Enable two factor authentication, PR #8079
- Internationalization of all user interface components
- Right-to-left implementation throughout user interface
- Internationalized Graph import and export, PR #7990
- Updates SPARQL endpoint to work with internationalized resources, PR #8063
- Internationalize JSONLD import export, PR #7953
- Write and load `.po` files, PR #8057  
- Consolidate languages into single table, PR #8153
- Import csv multiple string values by language
- Export csv in available languages, PR #8164
- Add internationalized graph caching, PR #8577
- Remove Collector Project Manager and CouchDB dependency, PR #8599

### Additional highlights

  
### Dependency changes
Note: Items prepended with `*` are depdency changes from 6.1.0 -> 6.2.0. Version 6.2.0 is currently un-released, will be released shortly after 7.0.0.

```
Python:
    Upgraded:
        Django 3.2.14 -> 3.2.15
        elasticsearch 7.11.0 -> 8.3.1
        *celery 4.4.4  -> 5.2.7
        *django-celery-results 1.2.1 -> 2.4.0
        *django-revproxy 0.9.15 -> 0.10.x
        *django-compressor 2.2 -> 3.1
        *django-libsass 0.7 -> 0.9

    Added:
        polib==1.1.1
        pyotp>=2.6.0
        qrcode>=7.3.1
        django-webpack-loader==1.5.0
        arches-django-revproxy==0.10.0

JavaScript:
    Upgraded:
        @mapbox/geojsonhint: 3.0.0 -> 3.0.1
        core-js: 2.5.7 -> 3.21.1
        codemirror: 5.24.0 -> 5.65.6
        requirejs-text: 2.0.12 -> 2.0.16
        underscore: 1.9.1 -> 1.13.4
        *bootstrap 3.3.7 -> 3.4.1
        *bootstrap-colorpicker 2.3.6 -> 2.5.3
        *codemirror 5.24.0 -> 5.65.6
        *datatables.net 1.10.22 -> 1.12.1
        *datatables.net-bs 1.10.12 -> 1.12.1
        *datatables.net-buttons 1.5.4 -> 2.2.3
        *datatables.net-buttons-bs 1.5.4 -> 2.2.3
        *datatables.net-responsive 2.1.0 -> 2.3.0
        *datatables.net-responsive-bs 2.1.0 -> 2.3.0
        *jquery 3.3.1 -> 3.6.1
        *jquery-migrate 3.0.0 -> 3.4.0
        *jquery-validation 1.17.0 -> 1.19.5
        *moment 2.10.5 -> 2.29.4
        *moment-timezone 0.5.0 -> 0.5.34
        *underscore 1.9.1 -> 1.13.4

    Added:
        @babel/runtime: 7.17.2
        cross-fetch: 3.1.5
        regenerator-runtime: 0.13.9
        webpack-bundle-tracker: 1.4.0
```

### Breaking changes
- Support for Arches Collector has been dropped and all references removed.
- `media/packages` has been renamed to `media/node_modules`, and this change should be reflected in `.yarnrc` and anywhere else `packages` is referenced.
- `python manage.py collectstatic` has been deprecated and replaced with `python manage.py build_production`
- If internationalization is important to your project, strings in component templates (eg. any template that is a dependency of a JavaScript file) should be moved to `javascript.htm`
- `STATIC_ROOT`, `STATIC_FILES`, `STATICFILES_DIRS`, and `STATICFILES_STORAGE` settings have been changed in the arches core application, may need to be handled for at the project level.
- `MOBILE_OAUTH_CLIENT_ID` has been renamed to `OAUTH_CLIENT_ID`
- The `{% load staticfiles %}` template tag must be replaced with `{% load static %}`
- The elasticsearch 8 upgrade requires installation specific [modifications](#elasticsearch)

### Upgrading Arches
1. You must be upgraded to at least version 6.1.0 before proceeding.

2. Remove django-revproxy `pip uninstall django-revproxy`

3. Be sure to backup your database before proceeding.

4. Upgrade to Arches 7.0.0
    ```
    pip install --upgrade arches
    ```

5. Within your project with your Python 3 virtual environment activated:

        python manage.py migrate
        python manage.py updateproject
        python manage.py es reindex_database

6. **Important:** If you are maintaining an Arches package, be sure to export your graphs and resource instance data before re-importing or reloading your package.
7. <a name="elasticsearch"></a> Elasticsearch 8 (ES 8) enables a feature called xpack security by default.  See the elasticsearch 8 documentation for details on the new features and how you can use them to secure your elasticsearch instance. 
    - By default, xpack security turns on SSL with a self-signed certificate and will expect requests to come from an ES user.  Elasticsearch users can be created via the elasticsearch-users command. Uncomment and override `ELASTICSEARCH_CONNECTION_OPTIONS` in `settings.py` line as needed (with an appropriate password and certificate settings) in your project settings or settings_local files.  
    - If you choose to turn off xpack security in your ES 8 instance (not recommended) you will need to override the `ELASTICSEARCH_HOSTS` setting and change the scheme to "http".

### Upgrading an Arches project
1. in your project's `settings.py` file:
    1. `MOBILE_OAUTH_CLIENT_ID` must be renamed to `OAUTH_CLIENT_ID`
    2. ensure `webpack_loader` has been added to `INSTALLED_APPS`
    3. ensure `APP_NAME` and `ARCHES_NAMESPACE_FOR_DATA_EXPORT` have been defined
    4. ensure the following attributes exist in the file:
        ```
        STATIC_ROOT = os.path.join(ROOT_DIR, "staticfiles")
        ```
        ```
        STATIC_URL = "/static/"
        ```
        ```
        STATICFILES_DIRS =  (
            os.path.join(APP_ROOT, 'media', 'build'),
            os.path.join(APP_ROOT, 'media'),
        ) + STATICFILES_DIRS
        ```
        ```
        WEBPACK_LOADER = {
            "DEFAULT": {
                "STATS_FILE": os.path.join(APP_ROOT, "webpack/webpack-stats.json"),
            },
        }
        ```

2. If you have not yet run `python manage.py updateproject`, do so now. It will create a `/webpack/` directory in your project and copy over the webpack configuration files from arches core.
    - `webpack/webpack-user-config.js` is git-ignored and is meant for user-specific webpack settings
    - `webpack/webpack-meta-config.js` is meant for project-specific webpack settings
    - ensure that `APP_ROOT_DIRECTORY` and `ARCHES_CORE_DIRECTORY` are pointing to the correct directories in your system in either `webpack/webpack-user-config.js`  or `webpack/webpack-meta-config.js` 

3. Copy `.eslintrc.js`, `.eslintignore`, `.babelrc`, `.browserslistrc` and `.stylelintrc.json` from arches-core to your project directory.

4. Update the project's `.gitignore` file to include:
    ```
    my_project/staticfiles
    my_project/webpack/webpack-stats.json
    my_project/webpack/webpack-user-config.js
    node_modules
    ```

5. Update the contents of `.yarnrc` to reference `node_modules` instead of `packages`:
    ```
    --install.modules-folder "./media/node_modules"
    --add.modules-folder "./media/node_modules"
    ```

6. The project's `package.json` file will need to be manually updated to include `devDependencies` and updated yarn `scripts`. It should contain the following `scripts` object and `devDependencies` object:
    ```
    "scripts": {
        "build_production": "NODE_PATH=./media/node_modules NODE_OPTIONS=--max_old_space_size=8192 NODE_ENV=production ./media/node_modules/.bin/webpack --config webpack/webpack.config.prod.js",
        "build_development": "NODE_PATH=./media/node_modules NODE_OPTIONS=--max_old_space_size=8192 ./media/node_modules/.bin/webpack --config webpack/webpack.config.dev.js",
        "build_test": "NODE_PATH=./arches/app/media/node_modules NODE_OPTIONS=--max_old_space_size=8192 ./arches/app/media/node_modules/.bin/webpack --config webpack/webpack.config.dev.js --env test=true",
        "start": "NODE_PATH=./media/node_modules NODE_OPTIONS=--max_old_space_size=8192 ./media/node_modules/.bin/webpack serve --config webpack/webpack.config.dev.js"
    },
    ```
    ```
    "devDependencies": {
        "arches-dev-dependencies": "archesproject/arches-dev-dependencies#dev/7.0.x"
    },
    ```

7. Update the templates in your project:
    - If a template is using the `{% load staticfiles %}` template tag, it must be updated to `{% load static %}`
    - If a template is using the `{% static '/path/to/file' %}` template tag, it must be updated to `{% webpack_static '/path/to/file' %}`
        - Be sure to include `{% load webpack_static from webpack_loader %}` at the top of any template where this change is made

8. The pattern used for JavaScript component template dependencies has changed, and any project components must be updated as well. Consider the
following example with our `reports/default.js` component:

    ```
    # dev/6.2.x

    define(['knockout', 'viewmodels/report'], function(ko, ReportViewModel) {
        return ko.components.register('default-report', {
            viewModel: function(params) {
                params.configKeys = [];

                ReportViewModel.apply(this, [params]);
            },
            template: { require: 'text!report-templates/default' }
        });
    });
    ```
    ```
    # dev/7.0.x

    define(['knockout', 'viewmodels/report' 'templates/views/report-templates/default.htm'], function(ko, ReportViewModel, defaultReportTemplate) {
        return ko.components.register('default-report', {
            viewModel: function(params) {
                params.configKeys = [];

                ReportViewModel.apply(this, [params]);
            },
            template: defaultReportTemplate
        });
    });
    ```

9. (optional) If internationalization is important to your project, strings in component templates (eg. any template that is a dependency of a JavaScript file) should be moved to `javascript.htm` and added as a key-value pair to an `arches-translations` HTML object. You must then update your component template to use the new string: 
    ```
    # my_project/my_project/templates/javascript.htm

    {% extends "javascript.htm" %}

    {% load static %}
    {% load l10n %}
    {% load i18n %}

    {% block arches_translations %}
    {{ block.super }}
    <div 
        class='arches-translations'
        my-key-name='{% trans "My key value." as myKeyValue %} "{{ myKeyValue|escapejs }}"'
    ></div>
    {% endblock arches_translations %}

    ```
    ```
    # my_project/my_project/templates/path/to/component.htm

    <span data-bind="text: $root.translations.textKeyOnArchesTranslationObject"></span>

    # OR

    <select 
        data-bind="placeholder: $root.translations.textKeyOnArchesTranslationObject"
    ></select>

    # OR

    <input data-bind="attr:{placeholder: $root.translations.textKeyOnArchesTranslationObject}></input>
    ```
    Note: Keys added to the `arches-translations` HTML object should be dash-delineated, eg. `my-key-name`. When referenced in a component template, the key should be camelcase instead, eg. `myKeyName`.

10. Update the project's `index.htm` `/package` references to `/node_modules` references, along with any other references to `/media/packaages` in your codebase.

11. Start your application server.

12. In a seperate terminal, navigate to root directory of the project ( you should be on the same level as `package.json`) 
    - manually remove the `yarn.lock` file, eg. `rm yarn.lock`
    - run `yarn`, this will install updated frontend dependencies in `/media/node_modules`.
    - (optional) You can remove `/media/packages`

13. In the same terminal window where you ran `yarn`, run either `yarn build_development` or `yarn_start`. This will generate a `media/build` directory.
    - `yarn start` will build the frontend of the application and then start a webpack development server
    - `yarn build_development` will build the frontend of the application

14. If you are running Arches on Apache, be sure to run:

    ```
    python manage.py build_production
    ```
    and restart your server.
    ```
    sudo service apache2 reload
    ```

15. Finally, If you are running Celery, you should also restart your Celery worker(s). The process for doing this depends on how Celery is being run.