VK4PK Maintaining the LyonsComputer Web Site

Contents:

Prerequisites
Setting up the Platform Environment
The bldidx.sh script
Setup the sitecopy configuration file
Using "checksum" State to Control Uploads
Initialising Sitecopy
scripts
The "bldidx.sh" Script echo "
$item_no_ext$SP4 [Home]" >>$OUTFILE echo "
$item$SP4 [Home]" >>$OUTFILE
Recovering from "sitecopy" Database Corruptions
Link Checkers
Usefull Linkchecker Commands
Trouble Shooting
References

[Top][Home]

Prerequisites

ARADIO enviorment variable set to the Amateur Radio Folder
$ARADIO/bin on Path
The application linkchecker and sitecopy must be installed
"bldisx.sh" and "h3.sh" scripts in $ARADIO/bin folder
Diskspace and permission to create temporary folder $HOME/www-build
Create and populated $HOME/.sitecopyrc

[Top][Home]

Setting up the Platform Environment

Add an environment variable, "ARADIO" to .bashrc file in the home directory

[Top][Home]

The bldidx.sh script

The bldidx.sh script requires linkchecker and sitecopy applications.

Run the bldidx.sh script to make a temporary copy of the html file tree to the ~/www-build folder. Only required html files and their dependant files (images, pdf's, etc) will be copied.

[Top][Home]

Setup the sitecopy configuration file

The files state is stored in the files in ~/.sitecopy folder and is used to discover when a file has been changed. Two methods are supported, and can be selected using the state option, with either parameter: "timesize" (the default), and "checksum".

"timesize" uses the last-modification date and the size of files to detect when they have changed.
"checksum" uses an MD5 checksum to detect any changes to the file contents.

Note that MD5 checksumming involves reading in the entire file, and is slower than simply using the last-modification date and size. It may be useful for instance if a versioning system is in use which updates the last-modification date on a 'checkout', but this doesn't actually change the file contents.
Source: "man sitecopy"

*** NOTE the .sitecopy can only be implemented on one workstation for the site state
to be current. ***

Setup sitecopy's configuration file, ~./sitecopyrc

cd # cd to $HOME # create the sitecopy config file mkdir -m 700 ~/.sitecopy # with permissions 700 # create the rcfile with site specific information # cat > ~/.sitecopyrc << 'EOF' # # .sitecopyrc - site specific information # site lyonscomputer.com.au server lyonscomputer.com.au username xxxxxxxxxx password xxxxxxxxxx local /home/glenn/www-build remote ~/httpdocs/ state checksum EOF chmod 0600 ~/.sitecopyrc # set permissions

[Top][Home]

Using "checksum" State to Control Uploads

Important Note: Because the bldidx script copies all web page files to a temporary build directory if the "timesize" method is used, sitecopy will copy every file to the live website every time. Therefore, the "checksum" method is required if only modified files are to be uploaded.

[Top][Home]

Initialising Sitecopy

Sitecopy must be initialises before use. This will create a status file in the ~/.sitecopy folder with the domain name of the website.

Initialise Sitecopy: mkdir ~/www-build # must exist even if empty for sitecopy to run. sitecopy --fetch lyonscomputer.com.au

Run bldidx.sh to populate the www-build folder with files. This will only be files required to run the web page. That is all files below index.html.

[Top][Home]

scripts

Two scripts have to be created.

bldidx.sh - Populates ~/www-build with only those files required for the web site.
h3.sh - create Contents from level 3 headings
h3-all.sh
bartab.sh - generate a table

Cut and paste the textareas below to $ARADIO/bin folder.

[Top][Home]

The "bldidx.sh" Script

#!/bin/bash
  # bldidx.sh - build a web site in $BLDDIR ready for export to live web server
  # 20180429gnl - Glenn Neil Lyons
  # Vers 3.0
  #
  # This script builds an index.html with the top level topics using the folders (under Amatuer-Radio)
  # as the main menu with links to the individual topic content section. The contect section lnk to
  # the individual content pages.
  #
  #

################################
  # Check Enviroment
  #################################
  # Check we are root
  if [ "$(id -u)" = "0" ];then
    echo "This script does not require to be run as root" 1>&2
    echo "Please login as the user that maintains the LyonsComputer Web Site."
    exit 1
  fi
  command -v linkchecker >/dev/null 2>&1 || { echo >&2 "'linkchecker' is not installed.  Try 'sudo apt install linkchecker'"; exit 1; }
  command -v sitecopy >/dev/null 2>&1 || { echo >&2 "'sitecopy' is not installed.  Try 'sudo apt install sitecopy'"; exit 1; }
  # must have $ARADIO as base folder of Amateur Radio html files
  if [ -z $ARADIO ]
  then
  	echo $0: "Enviorment Variable $ARADIO is not set".
  	exit 1
  fi
  cd $ARADIO   # must start here # Change to the working directory
  if [ "$1x" == "x" ];then
  	operation="build"	# do the full build
  else
  	operation=$1
  fi
  # operation="components"	# check components
  echo "Running in $operation mode".

##################################
  # Define Varaibles and Initialise
  ##################################
  INDEX="index.html"
  BLDDIR="/home/glenn/www-build"
  OUTFILE=$BLDDIR/$INDEX
  dirname=${file%/*}              # directory name
  LOGDIR="00-logs"
  LOG=$LOGDIR/"bldidx.log"
  DATE=`date`
  start=`date +%s`
  MISFILE=$ARADIO/$LOGDIR/"Missing-Files.txt"
  MISCNT=0  # count of missing file required to be loaded
  SKIPPED_PAGES="$ARADIO/$LOGDIR/Skipped_pages.txt"
  skipped_cnt=0  	# html pages without the "<title>VK4PK" string
  SP=" "
  SP2="$SP$SP"
  SP4="$SP$SP$SP$SP"
  INDENT=""	# increment with SP4 for each level down
  cnt=0		# count the html pages processed
  COMCNT=0	# count the comonent files required

#########################################################################################
  # Functions
  #########################################################################################

################################################################################################
  # Are we ready to go?
  ################################################################################################
  initialise_build() {
  echo -n "/home/glenn/www-build will be overwritten! PROCEED? [Y|y]"
  read ans
  if [ "$ans" != "Y" -a "$ans" != "y" ]
  then
  	echo "Quitting."
  	exit 1
  fi
  echo Removing $BLDDIR and all its files
  rm -rf $BLDDIR
  if [ ! -d $BLDDIR ];then
  	mkdir $BLDDIR
  fi
  # Copy stylesheet and setting permissions
  if [ -f stylesheet-for-amateur-radio.css ]
  then
          cp stylesheet-for-amateur-radio.css $BLDDIR
  else
          echo "WARNING - canot find file: stylesheet-for-amateur-radio.css"
  fi
  # Copy images files
  if [ -f images/LyonsLogo.png ]
  then
          mkdir $BLDDIR/images
          cp images/LyonsLogo.png $BLDDIR/images/LyonsLogo.png
  else
          echo "WARNING - cannot find file: images/LyonsLogo.png"
  fi
  # Initialise logs files
  cat /dev/null > $MISFILE
  cat /dev/null > $SKIPPED_PAGES
  }

#############################################################################################
  # Rotate Logs
  #############################################################################################
  rotate_log() {
  # rotate the log file
  echo "Rotating the Log file $LOG"
  for n in {8..1}
  do
          if [ -f $LOG.$n ]
          then
                  mv -f "$LOG.$n" "$LOG.$((n+1))"
          fi
  done
  if [ -f $LOG ]
  then
          mv $LOG $LOG.1
  fi
  echo "-----------------------------------------------------------------------------" >>$LOG
  echo "bldidx.sh $DATE">$LOG # Initialise a new log file
  return
  }

################################################################################################
  #
  ################################################################################################
  build_header() {
  cat > $OUTFILE << 'EOF'   # first output to file so reset file
  <!DOCTYPE html>
  <html>

<head>
    <meta charset="utf-8">
    <title>Amateur Radio Station VK4PK - Glenn</title>
    <link rel="stylesheet" type="text/css" href="stylesheet-for-amateur-radio.css">
    <link rel="icon" type="image/png" href="images/LyonsLogo.png">
  </head>

<body>
  <center><h1 id="home">VK4PK Menu</h1></center>
  
Contents:
<ol>
    <li><a href="#prerec">Prerequisites</a>
    <li><a href="#env">Setting up the Platform Environment</a>
    <li><a href="#bldidx.sh">The bldidx.sh script</a>
    <li><a href="#config">Setup the sitecopy configuration file</a>
    <li><a href="#sitecopy">Using "checksum" State to Control Uploads</a>
    <li><a href="#Init">Initialising Sitecopy</a>
    <li><a href="#scripts">scripts</a>
    <li><a href="#bldidx">The "bldidx.sh" Script</a>
echo "    <li><a href="# id=\"$item_no_ext\">$item_no_ext$SP4<a href=\"#home\">[Home]</a></a>" >>$OUTFILE
echo "    <li><a href="# id=\"$item\">$item$SP4<a href=\"#home\">[Home]</a></a>" >>$OUTFILE
    <li><a href="#bad">Recovering from "sitecopy" Database Corruptions</a>
    <li><a href="#online">Link Checkers</a>
    <li><a href="#cmd">Usefull Linkchecker Commands</a>
    <li><a href="#trouble">Trouble Shooting</a>
    <li><a href="#Ref">References</a>
</ol>
<br>
<a href="#top">[Top]</a><a href="../../index.html">[Home]</a>
<hr>

  -->
  EOF
  }

################################################################################################
  # Build the top level index Section. Add ".ignore" file to exclude the folder and it's contents
  ################################################################################################
  build_top_index() {
  cat >> $OUTFILE <<'EOF'
  <style>
      ul {
          columns: 3;
      }
  </style>
  <ul>

EOF
  for folder in `ls`
  do
          if [ ! -d $folder ];then # only want folders
                  continue
          fi
          if [ -f $folder/.ignore ];then    # we do not want this folder
                  continue
          fi
          echo "<li><a href="""#${folder}""">"${folder}"</a><br>" >>$OUTFILE
  done
  echo "</ul>" >>$OUTFILE
  echo "<br><br><br><br>" >>$OUTFILE
  echo "<hr>" >>$OUTFILE
  }

###########################################################################
  # Copy file Componects to the temporary build folder (www-build)
  ###########################################################################
  copy_components() {
  component=""
  dirname=${line%/*}              # directory name

# NOTE linkchecker only report missing links so this script relies on the the build directory not
  # having the linked files in place untill after the procedure is run
  #
  # tell linkchecker "-ocvs" to output in csv format
  # "2>" dump error messages
  # "sed '1,4d ; /^# /d' delete all lines 1 to 4 and all "#" comment lines
  #| cut -d';' -f1` set the delimiter to ";" and select the second field
  # linkchecker --no-warnings --no-status -ocsv $BLDDIR/$line 2>/dev/null |sed '/^#/d;/^urlname/d' | cut -d';' -f1
  #
  # for component in `linkchecker -ocsv $BLDDIR/$line 2>/dev/null| sed '1,5d ; /^# /d' | cut -d';' -f1`
  for component in `linkchecker --no-warnings --no-status -ocsv $BLDDIR/$line 2>/dev/null |sed '/^#/d;/^urlname/d' | cut -d';' -f1`
  do
  	NEEDS=$dirname/$component
  	echo "    requires-->"$NEEDS
  	echo "    requires-->"$NEEDS >>$LOG
  	let "COMCNT=COMCNT+1"
  	cp --parent $NEEDS $BLDDIR              # --parents creates parent folder same as source
  	if [ $? -ne 0 ]
  	then
  		let "MISCNT=MISCNT+1"
  		echo "$line --> $NEEDS" >> $MISFILE
  	fi
  done
  }

########################################################################
  # Build the contents part of the menu system to link html pages
  ########################################################################

build_contents() {

declare -a array
  cnt=0
  indent=""

cd $ARADIO

for line in `find * -name "*.html" -print | sort `
  do
  	if [ "$line" = "./index.html" ];then
  		continue		# skip
  	fi

if [ -f $line ] && [ ${line: -5} == ".html" ];then
  		if ! grep -q "<title>VK4PK" "$line" ;then
  			if [ "$operation" == "build" ];then
  				echo $line >> $SKIPPED_PAGES
  			fi
  			((skipped_cnt++))
  			continue		# skip
  		fi
  	fi

# Build an array of directories and files
  	list=`echo $line | sed 's/\// /g'`
  	item=""
  	((i=-1))
  	for item in $list
  	do
  		((i++))
  		array[$i]=$item
  	done

# remove repeat last heading if same as html filename
  	# DO not change $i or $item before this section
  	html_file=${item%.*}        # remove the extension part
  	second_last_item="${array[(($i-1))]}"
  	if [[ $second_last_item == $html_file ]];then
  		unset array[$i]			# remove the last
  		array[(($i-1))]=$item
  		unset last_array[$i]			# remove the last
  		last_array[(($i-1))]=$item
  	fi

# Loop through the array and output the appropriate menu lines
  	item=""
  	str=""
  	length=${#array[@]}
  	((length++))  		# Start at 1 not 0

for (( j=0; j<${length}; j++ ));
          do
  		item=${array[$j]}
  		# Strip off the heading of the section
  		if [[ $j == 0 ]];then
  			# gnl20220114 Special case with one html file/page in the top level directory
  			# The array structure needs to be rewritten but this will work around the poor design
  			# Print a section heding and then a link for the html page
  		        if [[ ${array[$j]} = ${last_array[$j]} ]];then
  				html_file=${item%.*}        # remove the extension part
  				extension="${array[$j]##*.}"
  				if [ $extension == "html" ];then
  					item_no_ext=${item%.*}		# Remove extension in this case
  					echo "<h3 id=\"$item_no_ext\">$item_no_ext$SP4<a href=\"#home\">[Home]</a></h3>" >>$OUTFILE
  					echo "$SP4<a href=\"$line\">$html_file</a><br>" >>$OUTFILE
  					cp --parent $line $BLDDIR
  					echo $line
  					echo $line >>$LOG
  					copy_components
  				fi
                          fi
  			# gnl20220114 - end

if [[ ${array[$j]} != ${last_array[$j]} ]];then
  				if [ "$operation" == "build" ];then
  					echo "<h3 id=\"$item\">$item$SP4<a href=\"#home\">[Home]</a></h3>" >>$OUTFILE
  				fi
  			fi
  		fi
  		if [ $j -gt 0 ];then
  			indent=$indent$SP4
  			# This is the link so copy file to www-build and get it's dependant files
  			if [[ "$item" == *".html" ]]; then
  				html_file=${item%.*}        # remove the extension part
  				if [ "$operation" == "build" ];then
  					echo "$indent<a href=\"$line\">$html_file</a><br>" >>$OUTFILE
  				fi
  				cp --parent $line $BLDDIR
  				echo $line
  				echo $line >>$LOG
  				copy_components
  			else
  				# Just a heading not a link
  				if [[ ${array[$j]} != ${last_array[$j]} ]];then
  					if [ "$operation" == "build" ];then
  						echo "$indent$item<br>" >>$OUTFILE
  					fi
  				fi
  			fi
  		fi
  	done

((cnt++))
  	#echo "$cnt $line"
  	### copy this array to be last_array in next iteration ###
  	last_array=("${array[@]}")
   	unset array  # need a clean array fo rnext iteration
  	indent=""

# for Testing Only
  # 	if [ $cnt -gt 50 ];then
  # 		exit
  # 	fi
  done
  }

###################################
  # Footer
  ###################################

build_footer() {
  echo "    </body>" >> $OUTFILE
  echo "</html>" >> $OUTFILE
  }

install_build() {
  echo "Setting permissions"
  chown -R glenn:glenn $BLDDIR
  chmod -R a+rw $BLDDIR
  echo "Returning a fresh copy on index.html to `pwd`"
  cp $BLDDIR/index.html .
  end=`date +%s`
  secs=$((end-start))
  echo "-----------------------------------------------------------------------------"
  echo $DATE
  echo "HTML Files:" $cnt
  echo "Component Files:" $COMCNT
  echo "Missing Files:" $MISCNT
  echo "Missing File List:" $MISFILE
  echo "Skipped Files: $skipped_cnt (html pages without the \"<title>VK4PK\" string)"
  echo "Skipped File List:" $SKIPPED_PAGES
  echo -n "Procedure Complete in "
  printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60))
  #
  # Write to log
  #
  echo $DATE >>$LOG
  echo "HTML Files:" $cnt >>$LOG
  echo "Component Files:" $COMCNT >>$LOG
  echo "Missing Files:" $MISCNT >>$LOG
  echo "Missing File List:" $MISFILE >>$LOG
  echo "Skipped Files:" $skipped_cnt >>$LOG
  echo "Skipped File List:" $SKIPPED_PAGES >>$LOG
  echo -n "Procedure Complete in " >>$LOG
  printf '%dh:%dm:%ds\n' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) >>$LOG
  }

#################################################################################
  # Print Info
  #################################################################################
  print_info() {
  if [ ${MISCNT} -gt 0 ]
  then
          echo "Missing File List:"
          cat $MISFILE
  fi
  echo "-----------------------------------------------------------------------------"
  if [ ${skipped_cnt} -gt 0 ]
  then
          echo "Skipped Pages File List:"
          cat $SKIPPED_PAGES
  fi
  echo "-----------------------------------------------------------------------------"
  }

upload_build_to_web() {

echo "sitecopy lyonscomputer.com.au     # to list changes to live site"
  echo "sitecopy --update lyonscomputer.com.au  # to sync the live site to local files"
  while true
  do
      read -p "Do you wish to run \"sitecopy --update lyonscomputer.com.au?\"" ANS
      case $ANS in
          [Yy]* ) sitecopy --update lyonscomputer.com.au ; break;;
          [Nn]* ) exit;;
          * ) echo "Please answer yes or no.";;
      esac
  done
  }
  ###################################################################################
  # End of fuction section
  ########################################################################################

if [ "$operation" == "build" ];then initialise_build ;fi

if [ "$operation" == "build" ];then  rotate_log  ;fi

if [ "$operation" == "build" ];then  build_header;fi

if [ "$operation" == "build" ];then  build_top_index;fi

if [ "$operation" == "build" ];then  build_contents;fi

if [ "$operation" == "build" ];then  build_footer;fi

if [ "$operation" == "build" ];then  install_build;fi

# print_info 	# Print the detailed logged files

if [ "$operation" == "build" ];then  upload_build_to_web; fi

#################### End of File #########################

[Top][Home]

Recovering from "sitecopy" Database Corruptions

If the the uploaded site becomes out of sync with the local database run the commands:

This can occur when an sitecopy update is interrupted or fails to complete. The fetch option can take some time as it has to recalculate the checksum for every file and folder on the server and update the production servers sitecopy database. The update may have to be run a few times to complete the process.

[Top][Home]

Link Checkers

W3C - Check links and anchors in Web pages or full Web sites:
https://validator.w3.org/checklink

LinkChecker - Check websites for broken links:
https://wummel.github.io/linkchecker/other.html

[Top][Home]

Usefull Linkchecker Commands

[Top][Home]

Trouble Shooting

If bldidx.sh script does not produce an updated website in ~/www-build as expected it is often because the script cannot handle files with spaces. Run $ARADIO/bin/ls-all-html-files.sh and inspect the output for loine with spaces.

[Top][Home]

References

sitecopy - maintain remote copies of web sites:
Man page: https://linux.die.net/man/1/sitecopy

Maintaining remote Websites with Sitecopy on Ubuntu 18.04
https://www.howtoforge.com/tutorial/maintaining-remote-web-sites-with-sitecopy-on-ubuntu-18-04/

[Top][Home]

Glenn Lyons VK4PK
glenn@LyonsComputer.com.au
Ver:gnl20181201 - pre published v0.9