Setting up a development environment

Quick setup of local development environment with SQLite

cd ~
git clone https://github.com/saltcorn/saltcorn.git
cd saltcorn
npm install --legacy-peer-deps && npm run tsc
PATH=$PATH:~/saltcorn/packages/saltcorn-cli/bin
npm run tsc; while [ 1 ]; do SQLITE_FILEPATH=~/sctestdb saltcorn serve --dev;done

This assumes you have node and npm already installed. If there are errors diring the npm install step, you may be missing systems packages.

The last command will run saltcorn serve in development mode. This will watch the files in the checked out repository and if you edit and save one of the files in the repository (or in any installed local modules; see below) the server will exit, recompile with tsc and restart. It will not automatically reload any pages that are open in a local browser so you will have to do that yourself.

Quick setup of plugin for local development

Make sure you have a development environment working as described above.

Then clone the plugin source repository with git:

  • If you are working with an existing plugin, clone the repository for the plugin to a directory, for instance in your home directory.
  • If you are creating a new plugin, create the repository on GitHub, clown this to a directory and create the package.json and index.js files, likely starting with copying these files from another plugin that is close in functionality to what you are developing.

Now you need to add this plugin to your Saltcorn instance. Run this command, in the saltcorn directory:

saltcorn install-plugin -d /path/to/plugin

The last argument is the absolute path to the directory where you checked out the plugin with git.

Then start saltcorn in development mode as usual:

npm run tsc; while [ 1 ]; do SQLITE_FILEPATH=~/sctestdb saltcorn serve --dev;done

If you already had Saltcorn running in a different terminal, stop it with Ctrl-C and restart.

Now the saltcorn development server should restart every time you save a file in the plug-in checkout directory, and you can start editing the code. If you add a new file you may need to add it to git to get the automatic restarts. You can always restart manually by pressing Ctrl-C and re-running.

Requirements

Attention! Master branch already uses Node.js v16. Legacy instructions used for Node.js v14 are marked like this for now. Will be removed from the guide later.

You need:

  • Node.js 16. Master branch already uses Node.js v16. 
  • You may need some system packages. On Ubuntu 20.04 these packages are required or at least helpful: 
    • libpq-dev build-essential python-is-python3 postgresql-client git libsystemd-dev entr
    • on some other distributions, you need python3 instead of python-is-python3
  • Lerna. Install with npm install -g lerna
    • In master branch lerna already replaced by using node workspaces . So you dont need to install lerna.
  • Nodemon. Install with npm install -g nodemon

Checkout and Install

Fork the Saltcorn repository (https://github.com/saltcorn/saltcorn) and checkout with ssh:

git clone [email protected]:{GITHUB_USERNAME}/saltcorn.git

now go to the saltcorn checked out repository and run

lerna bootstrap

npm install --legacy-peer-deps

npm run tsc

to prepare the local build. You also need to run this every time you add a new dependency to one of the packages.

You should add /path/to/saltcorn/packages/saltcorn-cli/bin to your PATH. if you use bash, you can run

echo 'export PATH=$HOME/saltcorn/packages/saltcorn-cli/bin:$PATH' >> ~/.bashrc
source ~/.bashrc

you should now be able to run the saltcorn command.

tomn@Toms-Mac-mini saltcorn % saltcorn

Command-line interface for Saltcorn, open-source no-code platform

 

VERSION

  @saltcorn/cli/0.5.6-beta.3 darwin-x64 node-v14.16.1

 

USAGE

  $ saltcorn [COMMAND]

 

COMMANDS

  add-schema       Add Saltcorn schema to existing database

  backup           Backup the PostgreSQL database to a file with pg_dump or zip

  create-tenant    Create a tenant

  create-user      Create a new user

  ...

If you don't see this, then something has gone wrong.

Setup

You also need to tell Saltcorn how it will connect to a database. You can do this just by running

saltcorn setup

Select the development option which will use SQLite. Now you should be able to run with

saltcorn serve

Connect to http://localhost:3000 and set up your account

Development server

You are now ready to edit any of the files in the saltcorn repository. When you save a file, the server does not automatically pick up changes. You can press Ctrl+C to restart it, and then you should see your changes if you read out the page in the browser.

The nodemon tool make this process a bit easier. It can watch for file changes and restart the server automatically when it sees changes. To run the Saltcorn server with nodemon, run

nodemon  `which saltcorn` serve

Saltcorn uses one worker process per CPU core. This may slow down start-up if you have a lot of cores. To reduce the number of worker processes, set the SALTCORN_NWORKERS environment variable

SALTCORN_NWORKERS=2 nodemon  `which saltcorn` serve

If you launch Saltcorn like this, you only have to save your file and reload the page. You may have to wait a few seconds before you are able to reload the page.

Development server with PostgreSQL

In order to connect your development set up to a PostgreSQL server, you should:

  • Install PostgreSQL
  • Create a user in PostgreSQL with a password (or set the postgres user password and use postgres user)
  • Edit the saltcorn configuration file. To find out where this is, type saltcorn info and look for the line configFilePath
  • The configuration file should look like this:

{
        "host":"localhost",
        "port":5432,
        "database":"saltcorn",
        "user":"postgres",
        "password":"foobar",
        "session_secret":"jjtrjtyjd",
        "multi_tenant": true
}

Running tests

Tests are run using the run-tests command in the saltcorn cli command. This is optionally followed by a package name, in which case only the tests for that package will be run. If no package is specified, then the test for all packages are run. Note that the package name here is the directory name in packages/, not the formal NPM package name (e.g. saltcorn-data, not @saltcorn/data)

The tests are run against a real database, which needs to be reset multiple times during the test runs. The implications of this for your ongoing application development depends on which database backend you are using.

  • If you are using the SQLite database, it will use whatever the file /tmp/sctestdb as the database. It will not affect applications in different files
  • You can at any time switch to a SQLite-based test runner - whenever the SQLITE_FILEPATH environment variable is set, Saltcorn will use the SQLite backend, even if the configuration file is set up to use PostgreSQL. Environment variables in general overwrite the configuration file. SQLITE_FILEPATH=/tmp/sqlite.db saltcorn run-tests saltcorn-data
  • If Saltcorn is configured to use PostgreSQL, it will switch to connect to the saltcorn_test database, which must be present on the same database and accessed with the same credentials as those configured in your configuration file or environment variables. Running the tests will therefore not affect any applications you are developing as these will be located in a different database. 

In order for the tests related to multi tenancy to work, you should have the following lines in your /etc/hosts file:

127.0.0.1 example.com sub.example.com sub1.example.com sub2.example.com sub3.example.com sub4.example.com sub5.example.com
127.0.0.1 otherexample.com

The end to end test require the chromium browser to be installed. If it is in a different location than /usr/bin/chromium-browser, you can set the PUPPETEER_CHROMIUM_BIN environment variable.

There is a vagrant image that runs the full test suite against a check out from GitHub: https://github.com/saltcorn/saltcorn/tree/master/deploy/vagrant-test-install/run-tests. To use this file, install vagrant, cd to deploy/vagrant-test-install/run-tests and run vagrant up

Developing the builder

The drag-and-drop builder used to build pages and views is a React component which is compiled separately. Its source package is in packages/saltcorn-builder in the saltcorn repository. The build artefact is checked into version control and part of the saltcorn-builder NPM package as dist/builder_bundle.js. This is done to avoid needless rebuilds and to reduce the size of the node_modules installed. In the saltcorn-builder package, the build npm script can be used to rebuild this bundle, which is imported and statically served by the Saltcorn server process

To rebuild the builder bundle for the first time,

cd packages/saltcorn-builder

npm install

npm install [email protected]

npm run build

This regenerates dist/builder_bundle.js. Subsequent builds can be performed with

npm run build

You can also perform a development build, which is faster to build, gives you better error messages but a larger bundle

npm run builddev

Running either the production or the development build will be picked up if you are running a nodemon saltcorn server in a different terminal window.

In order to get continuous builds, without needing to re-run the build script manually, you can use the entr tool (installed through your OS package manager):

git ls-files | entr npm run builddev

in the saltcorn-builder directory. If you run the nodemon process in one terminal window and the above command in a different terminal window, you should see any changes after saving a source file and reloading the page. This is a little bit slower as both the rebuild needs to finish and the server need to reload.