VK4PK Maintaining the LyonsComputer Web Site

Contents:
  1. Prerequisites
  2. Setting up the Platform Environment
  3. The bldidx.sh script
  4. Setup the sitecopy configuration file
  5. Using "checksum" State to Control Uploads
  6. Initialising Sitecopy
  7. Troubleshooting
  8. scripts
  9. The "bldidx.sh" Script
  10. Recovering from "sitecopy" Database Corruptions
  11. Link Checkers
  12. Usefull Linkchecker Commands
  13. Trouble Shooting
  14. References

[Top][Home]

Prerequisites

  1. ARADIO enviorment variable set to the Amateur Radio Folder
  2. $ARADIO/bin on Path
  3. The application linkchecker and sitecopy must be installed
  4. "bldisx.sh" and "h3.sh" scripts in $ARADIO/bin folder
  5. Diskspace and permission to create temporary folder $HOME/www-build
  6. Create and populated $HOME/.sitecopyrc
[Top][Home]

Setting up the Platform Environment

Add an environment variable, "ARADIO" to .bashrc file in the home directory



[Top][Home]

The bldidx.sh script

The bldidx.sh script requires linkchecker and sitecopy applications.

Run the bldidx.sh script to make a temporary copy of the html file tree to the ~/www-build folder. Only required html files and their dependant files (images, pdf's, etc) will be copied.



[Top][Home]

Setup the sitecopy configuration file

The files state is stored in the files in ~/.sitecopy folder and is used to discover when a file has been changed. Two methods are supported, and can be selected using the state option, with either parameter: "timesize" (the default), and "checksum".

"timesize" uses the last-modification date and the size of files to detect when they have changed.
"checksum" uses an MD5 checksum to detect any changes to the file contents.

Note that MD5 checksumming involves reading in the entire file, and is slower than simply using the last-modification date and size. It may be useful for instance if a versioning system is in use which updates the last-modification date on a 'checkout', but this doesn't actually change the file contents.
Source: "man sitecopy"

*** NOTE the .sitecopy can only be implemented on one workstation for the site state
to be current. ***

Setup sitecopy's configuration file, ~./sitecopyrc



[Top][Home]


Using "checksum" State to Control Uploads

Important Note: Because the bldidx script copies all web page files to a temporary build directory if the "timesize" method is used, sitecopy will copy every file to the live website every time. Therefore, the "checksum" method is required if only modified files are to be uploaded.


[Top][Home]

Initialising Sitecopy

Sitecopy must be initialises before use. This will create a status file in the ~/.sitecopy folder with the domain name of the website.

Initialise Sitecopy:

Run bldidx.sh to populate the www-build folder with files. This will only be files required to run the web page. That is all files below index.html.


[Top][Home]


Troubleshooting

Synchronization Problems In some circumstances, the actual files which make up the remote site will be different from what sitecopy thinks is on the remote site. This can happen, for instance, if the connection to the server is broken during an update.

When this situation arises, Fetch Mode should be used to fetch the list of files making up the site from the remote server.

If the site update does not update all the correct files the local checksum file maybe out of sync the actual site checksums.

If you do not have a local copy of the remote site, or the checksum file is out of sync, then you can use fetch mode to discover the state of the remote site.

"sitecopy --fetch sitename" to fetch the remote site's state.

NOTE sitecopy --synch sitename downloads a local copy of the remote site. Do NOT do this as the objective os the do the opposite, uplaod the local copy to the remote site.

NOTE NOTE NOTE Noe of the above worked when I accendently started a seconf bldidx.sh process when another was running. DO not understand why the --fetch did not get the difference. The www-build re-built correctly but would not update the remote site!!!

In the end I deleted index.html on the remote site and copy a fresh copy from www-build to the remote site using Filezilla



[Top][Home]

scripts

Two scripts have to be created.
  1. bldidx.sh - Populates ~/www-build with only those files required for the web site.
  2. h3.sh - create Contents from level 3 headings
  3. h3-all.sh
  4. bartab.sh - generate a table
Cut and paste the textareas below to $ARADIO/bin folder.

[Top][Home]

The "bldidx.sh" Script

Script to builld website:
bldidx.sh - build and upload the LyonsComputer.com.au website

[Top][Home]

Recovering from "sitecopy" Database Corruptions

If the the uploaded site becomes out of sync with the local database run the commands:


This can occur when an sitecopy update is interrupted or fails to complete. The fetch option can take some time as it has to recalculate the checksum for every file and folder on the server and update the production servers sitecopy database. The update may have to be run a few times to complete the process.

[Top][Home]


Link Checkers

W3C - Check links and anchors in Web pages or full Web sites:
https://validator.w3.org/checklink

LinkChecker - Check websites for broken links:
https://wummel.github.io/linkchecker/other.html

[Top][Home]

Usefull Linkchecker Commands



[Top][Home]

Trouble Shooting

If bldidx.sh script does not produce an updated website in ~/www-build as expected it is often because the script cannot handle files with spaces. Run $ARADIO/bin/ls-all-html-files.sh and inspect the output for loine with spaces.



[Top][Home]

References

sitecopy - maintain remote copies of web sites:
Man page: https://linux.die.net/man/1/sitecopy

Maintaining remote Websites with Sitecopy on Ubuntu 18.04
https://www.howtoforge.com/tutorial/maintaining-remote-web-sites-with-sitecopy-on-ubuntu-18-04/

[Top][Home]
Glenn Lyons VK4PK
glenn@LyonsComputer.com.au
Ver:gnl20181201 - pre published v0.9