This tool is still in active development (see below!) and is by no means the finished product - it's a continually evolving skunk works project. Apologies if things break from time to time! It is intended for only the hardiest of early adopters to get a feel for. Any feedback or suggestions will be gratefully received!

What's new?

Update: 16/1/19
Dashboard: Added extra analysis for the following inline tags <audio>, <bdi>, <br>, <button>, <canvas>, <data>, <datalist>, <del>, <embed>, <iframe>, <input>, <label>, <map>, <mark>, <meter>, <noscript>, <object>, <output>, <picture>, <progress>, <ruby>, <s>, <script>, <select>, <slot>, <svg>, <template>, <textarea>, <u>, <video>, <wbr>
Dashboard: Added extra analysis for the following block tags <dl>, <details>, <dialog>, <div>
Dashboard: New report that shows pages that do not have a <h1> present on them.
Update 8/1/19
UI: Updated DataTables library to V1.10.19.
Performance: Drastically improved the performance of Robots.txt comparisons by cutting out needless HTTP calls.
Crawl: When assets are checked (e.g. images, CSS, scripts) ad indication of progress is now given e.g. image 45 of 987. This is important for big sites as audits can appear stuck, when in fact they are not.
Crawl: When external links are checked an indication of progress is now given e.g. link 1 of 33.
Crawl: Added more progress messaging for the external link reports which can take a while to run.
Back-end: Created data structure for holding Skipped paths.
Dashboard: Created reports to show which links have been skipped and why.
Update 7/1/19
Back-end: Integrated Tideways XHProf Extension so that performance of the application can be better analysed.
Server: Setup XHGUI so that performance can be analysed.
Status: Updated with 2019 reports.
Status: Improved page loading performance by 50%.
Back-end: Improved DB performance by adding new indexes.
Update 6/12/18
Server: Increased root drive disk space by 50%.
Update 30/11/18
Dashboard: Fixed bug with image generation.
Upgraded to latest version of NodeJS V10
Update 27/11/18
Status: Added mechanism to clear the queued audits.
Update 23/10/18
Swapped out php-domain-parser library for the PHP League URI library
Update 22/10/18
Added setup toggle for AWS Rekognition & Comprehend - it's quite expensive in this setting!
Update 16/10/18
Integrated AWS Rekognition.
Created data structure for holding AWS Rekognition output values for Detected Labels.
Created data structure for holding AWS Rekognition output values for Faces.
Created data structure for holding AWS Rekognition output values for Text.
Improved capture and storage of 'data-src' attributes.
Capturing AWS Rekognition return values for Labels on images over 400 pixels wide (in order to get more meaningful results).
Capturing AWS Rekognition return values for Faces on images over 400 pixels wide (in order to get more meaningful results).
Capturing AWS Rekognition return values for Text on images over 400 pixels wide (in order to get more meaningful results).
Generated new report for AWS Rekognition of Objects / Labels discovered within images.
Generated new report for AWS Rekognition of Top Objects / Labels discovered within images.
Generated new report for AWS Rekognition of Text discovered within images.
Generated new report for AWS Rekognition of Top Text discovered within images.
Update 15/10/18
Integrated AWS Comprehend.
Created data structure for holding AWS Comprehend output values for Dominant Language.
Created data structure for holding AWS Comprehend output values for Entities.
Created data structure for holding AWS Comprehend output values for Key Phrases.
Created data structure for holding AWS Comprehend output values for Sentiments.
Capturing AWS Comprehend return values for Sentiment on page content within the <body> tag.
Capturing AWS Comprehend return values for Dominant Language on page content within the <body> tag.
Capturing AWS Comprehend return values for Entities on page content within the <body> tag.
Capturing AWS Comprehend return values for Key Phrases on page content within the <body> tag.
Generated new report for AWS Comprehend Sentiment for full page content within the <body> tag.
Generated new report for AWS Comprehend Entities for full page content within the <body> tag.
Generated new report for AWS Comprehend Entities by Type for full page content within the <body> tag.
Generated new report for AWS Comprehend Entity Occurrences for full page content within the <body> tag.
Generated new report for AWS Comprehend Key Phrases for full page content within the <body> tag.
Generated new report for AWS Comprehend Key Phrase Occurrences for full page content within the <body> tag.
Update 09/10/18
Upgraded Highcharts to 6.1.4
Update 05/09/18
Upgraded Bootstrap to 4.1.3
Update 12/06/18
Crawl: Fixed bug that prevented some single page crawls from being captured correctly.
Crawl: Fixed bug that sometimes occurred when the get_headers() PHP command failed.
Update 22/05/18
Dashboard: Fixed bug that was preventing the retrieval of site headers when they have an improperly configured SSL certificate.
Update 21/05/18
Downloads: Fix where the page base was required.
Crawl: Improved breaking of long strings for the UI during the crawl process.
Update 19/05/18
Setup: Fixed bug by improving the progress reporting final state, as in some cases the report said it was ready, before it actually was.
Dashboard: Fixed bug that sometimes resulted in an old dashboard report being returned instead of the latest version.
Update 18/05/18
Dashboard: Fixed bug that was truncating content from some reports.
Update 17/05/18
Dashboard: Added 20 new tags to the deprecated tag report.
Update 16/05/18
Crawl: Now capturing block level tag information for <address>, <blockquote>, <dd>, <dt>, <figcaption>, <figure>, <hr>, <li>, <p>, <pre>, <table>, <caption>, <legend>, <video>.
Dashboard: New report for content block tag summary.
Update 15/05/18
Crawl: Fixed bug that prevented deprecated tags been collected properly.
Dashboard: Added four new tags to the deprecated report, namely <embed>, <listing>, <plaintext> & <xmp>
Dashboard: New report for inline tag summary.
Dashboard: Extended inline tag analysis to cover the following tags <strong>, <em>, <b>, <i>, <abbr> <acronym>, <bdo>, <big>, <cite>, <code>, <dfn>, <kbd>, <q>, <samp>, <small>, <span>, <sub>, <sup>, <time>, <tt>, <var>
Back-end: Created data structure to support block level content element capture.
Update 14/05/18
Crawl: Fixed bug that was preventing crawl cancelling from occurring properly.
Content: Updated the features list page.
Update 11/05/18
Dashboard: New report for Folder structure of Linked Assets & Documents.
Dashboard: New report for Folder structure of Images.
Dashboard: New report for Folder structure of Scripts.
Dashboard: New report for Folder structure of Stylesheets.
Update 10/05/18
Dashboard: Fixed bug with "External links on same domain" report that didn't handle double sub-domains e.g.
Update 01/05/18
Upgraded Bootstrap to 4.1.1
Dashboard: Added new report which shows image assets that have been blocked by the Robots.txt.
Update 30/04/18
Back-end: Upgraded the AWS SDK to version 3.55.3.
Dashboard: Added new report which shows URL's that have been blocked by the Robots.txt.
Update 28/04/18
Dashboard: Improved the UX of the anon user journey, and those sites with small numbers of pages.
Update 27/04/18
Dashboard: Added new support that summarises the special tag counts.
Dashboard: Added new support that summarises the deprecated tag counts.
Dashboard: Fixed bug that was preventing the special tag reports from showing.
Dashboard: Added Country information to many of the external link reports.
Dashboard: Fixed bug on the biggest images report that showed duplicate results.
Status: Fixed bug that resulted in duplicated crawl report information.
Update 22/04/18
Dashboard: Added new report for listing any cookies detected during the initial page redirection process.
Update 20/04/18
Dashboard: Added better table captions for the meta tag reports.
Dashboard: Added new report for detected telephone numbers.
Dashboard: Added new report for detected skype numbers.
Dashboard: Added new report section for showing cookie analysis.
Dashboard: Added new report for showing all cookies.
Dashboard: Added new report for cookie 'Domain' parameter analysis
Dashboard: Added new report for cookie 'Path' parameter analysis
Dashboard: Added new report for cookie 'Name' parameter analysis
Dashboard: Added new report for cookie 'Value' parameter analysis
Dashboard: Added new report for cookie 'Expires' parameter analysis
Dashboard: Added new report for cookie 'Max Age' parameter analysis
Dashboard: Added new report for cookie 'Secure' parameter analysis
Dashboard: Added new report for cookie 'HTTP Only' parameter analysis
Back-end: Created data structure to support telephone numbers.
Back-end: Created data structure to support skype numbers.
Back-end: Created data structure to support cookies.
Back-end: Implemented cookie parsing functionality.
Back-end: Now storing all detected cookies in DB.
Update 19/04/18
Dashboard: Added new report for Robots.txt to show allow and disallow rules.
Dashboard: Added descriptions for newly detected header types.
Dashboard: Added IP addresses to some of the external domain reports.
Dashboard: Fixed bug introduced by PHP 7.2 that prevented some of the count values showing on the reports.
Dashboard: Added new report that shows external links on same domain (i.e. subdomains discovered off the primary top level domain).
Back-end: Created data structure to support pages / assets blocked by the Robots.txt file.
Back-end: Re-factored code within the crawl library so that it is more in-line for coding standards.
Update 18/04/18
UI: Upgraded to Highcharts 6.1.0.
Update 15/04/18
Dashboard: Added more table caption content.
Update 14/04/18
Back-end: Improved error handling around the dns_get_record() function.
Dashboard UI: Fixed row class name bug, preventing individual cell background colour overrides.
FIX Dashboard UI: Improved UI for showing allow / disallow rules on the robots table.
Update 13/04/18
Score: Factored Twitter Cards and Open Graph tags into the score.
Update 09/04/18
Upgraded Bootstrap to 4.1.0
Update 06/04/18
Dashboard: Extended the HTTP status code reports to include the number of times each url has been discovered.
Dashboard: Developer Helpers for redirects extended to include downloads for ONLY those pages that return a HTTP 404 status code.
Crawl: Created data structure to support both ALLOW and DISALLOW robots.txt rules.
Dashboard: Now showing ALLOW and DISALLOW robots.txt rules.
Update 30/03/18
Dashboard: Improved render of the Data Layer report.
Update 29/03/18
Thumb generation: Fixed issue that stopped thumbnail being generated on the first audit of a new site.
Dashboard: Created new report to output the contents of the Data Layer object that is discovered on each page.
Crawl: Created new data structure to support the capture of Google Tag Manager Data Layer tags.
Update 27/03/18
Dashboard: Updated meta description length recommendations from 160 characters to 300.
Update 21/03/18
Status: Added progress status column to overview table.
Update 19/03/18
Dashboard: Added HEX values to the Invalid URI report.
Update 17/03/18
Dashboard: New images report for Apple Touch Icons.
Dashboard: New images report for Apple Touch Icon Precomposed.
Dashboard: New images report for Mask Icons.
Dashboard: New internal link report for rel="revision" link attributes.
Update 13/03/18
Dashboard: Improved the rendering of the report at lower resolutions.
Dashboard: Added headers into the LHN to make navigation clearer.
Dashboard: Fixed bug that calculated the total weight of a given asset type by page incorrectly.
Dashboard: Improved copy throughout and ensured that each section has a full intro description.
Dashboard: Added 'see all' button above each table to pull in additional results.
Dashboard: Fixed bug with exporting the page rank data.
Dashboard: Fixed bug with showing all keyword analysis data.
Dashboard: Fixed bug with showing all rows across a variety of tables.
Update 12/03/18
Back-end: Upgrade to Adminer 4.6.2.
Performance: Refactored how and at what point Indexes are applied to the DB, resulting in an improved overall audit speed.
Back-end: Applied a busyTimeout to the DB to prevent database locking issues which occasionally occur.
Update 11/03/18
Setup: Improved layout and default settings.
Update 06/03/18
Dashboard: New report that shows analysis on the x_drupal_dynamic_cache HTTP Header.
Dashboard: Revised some table heading / column names to make the contents clearer.
Dashboard: Fixed full modal window bug introduced through upgrade to Bootstrap 4.
UI: Updated various styles to take better advantage of Bootstrap 4 styles, particularly for buttons.
Crawl: Now capturing the data-src attribute on images which is sometimes used for lazy-loading.
Update 28/02/18
Dashboard: Added view source to link
Update 23/02/18
Dashboard: Slanted table header bug fix introduced during Bootstrap 4 upgrade
Update 21/02/18
Back-end: Deprecated PhantomJS from the application in favour of Google Chrome.
Back-end: Refactored all usages of PhantomJS with Google Chrome equivalents.
Back-end: Website screenshots now only re-generate a maximum of once every 24 hours.
Update 20/02/18
Dashboard UI: Improved tables.
Dashboard: Added AMP detection and reporting.
Back-end: Integrated Google Chrome Headless
Dashboard: Added ability to take a full page Screenshot of any discovered page on a report, at 3 different resolutions.
Update 19/02/18
JS: Upgraded HighCharts to 6.0.7.
Update 06/02/18
Setup: Added pages 'at current depth' metric.
Update 01/02/18
Crawl: Added further exclusions for external domain checking.
JS: Upgraded HighCharts to 6.0.5.
Back-end: Fixed bug with IP determination of a user.
UI: Refactored the 'my reports' listing code so that it is more efficient, and can be used elsewhere.
UI: Fixed main navigation bug on mobile.
Login: Fixed issue when you attempt to login or logout from a non-root path page.
Dashboard: Added further IP analysis of website.
Update 31/01/18
Dashboard UI: Added messaging around the score, indicated its reliability.
Update 27/01/18
Dashboard UI: Copy and ordering tweaks throughout.
UI: Bootstrap v4 updates throughout.
Update 23/01/18
Dashboard UI: Added a dropdown text to each table that shows the options that are available for it.
Dashboard UI: Fixed issue with share button disappearing after viewing more.
UI: Upgraded ZeroClipboard library to v2.3.0
UI: Upgraded clipboard.js library to v1.7.1
UI: Upgraded bootstrap-tour library to v0.12.0
UI: Upgraded D3 to v4.12.2
UI: Integrated Vega v3.0.9
Update 21/01/18
UI: Added the dedicated Glyphicons library to the project.
UI: MAJOR upgrade of Bootstrap to V4 from V3.
UI: Tweaks throughout to take advantage of new approaches within V4.
UI: Upgraded Jqeury to V3.3.1
UI: Upgraded tablesorter jquery plugin to V2.29.3 and refactored the integration.
Dashboard: Fixed page anchor links on the overview screen.
Dashboard: Fixed page anchor links on the score screen.
Update 18/01/18
Server: Big stability improvement for larger crawls. Achieved full analysis for crawl with 20,000 pages, and 300Mb of scraped linking data.
Dashboard: Excluded RSS feeds from size and performance metrics.
Dashboard: Added new developer helper report for nginx redirects.
Update 12/01/18
Dashboard: Fixed bug with Page Rank calculation to exclude factoring in RSS links.
Update 09/01/18
Dashboard: Fixed anchor mappings between the left hand navigation items, and the reports that had become mis-aligned.
Dashboard: Another fix for checking images, scripts and stylesheets that are behind HTTP Auth.
Dashboard: Fix for checking robots.txt and sitemaps that are behind HTTP Auth.
Dashboard: Updates to stylesheets and scripts reports to cater for additional ways in which they can be embedded into a page.
Dashboard: Created new dedicated meta description report
Update 05/01/18
Dashboard: Started the work for the Drupal specific reports.
Update 04/01/18
Dashboard: Fix for single view where it was not possible to show all results.
Dashboard: Fix for checking images, scripts and stylesheets behind HTTP Auth.
Dashboard: Reporting on those images, scripts and stylesheets that are behind HTTP Auth.
Update 24/12/17
Install: Prevented the need to import the GeoIP data when developing locally.
Update 21/12/17
Back-end: Fixed bug when extra data was being fetched by AJAX, including the slanted layout.
Visualisations: Improved rendering on lower resolution devices.
Visualisations: Added new default visualisation showing the language distribution.
Dashboard: Fixed several reports that needed to be restricted to analysis only on pages that return a HTTP 200 code.
Setup: Added HTTP status code feedback to the audit progress modal window.
Back-end: Improved how the progress of an audit is reported back-end. Update calls are now lighter and less frequent.
Setup: Added pages per second metric to the reporting progress
Update 20/12/17
Back-end: Updated various code functions to support PHP 7.2 inc usages of each() and call_function().
Server: Upgraded php from 7.1 to 7.2.
Dashboard: New duplicate content report for heading content.
Dashboard: Various UI fixes.
JS: Upgraded D3 library to 4.12.0
Update 19/12/17
UI: Fixed bug with My Reports section.
Back-end: Upgraded hashing to use Murmur3 instead of SHA1.
Dashboard: New duplicate content report for paragraph HTML content.
Dashboard: New duplicate content report for paragraph text excluding HTML tags.
Update 18/12/17
JS: Upgraded HighCharts to 6.0.4.
Update 17/12/17
Status: Optimised page loading time.
Dashboard: New duplicate content report for body HTML.
Dashboard: New duplicate content report for body excluding HTML tags.
Crawl: Capture the SHA1 hash of various parts of the DOM to later use to determine if there is duplicate content.
Crawl: Created new data structure to support duplicate content.
Update 15/12/17
Back-end: Closed DB connections that were left lingering with AJAX calls.
Server: Upgraded nginx from 1.10 to 1.13.
Update 12/12/17
Back-end: Improved the way in which files are written to disk for crawls.
UI: Increased report loading performance by remotely fetching required JSON data from flat file, rather than embedding it in the report markup.
Bug: Fixed bug with Page Rank calculation
Update 27/11/17
UI: Enabled the Highcharts Boost module as the Sunburst visualisations were suffering performance problems when rendering 1000+ data points.
Update 26/11/17
UI: Updated DataTables library to V1.10.16.
Update 25/11/17
Crawl: Fixed bug with text analysis.
Newsletter sign-up: Fixed bug with inserting new records.
Crawl: Fixed bug that sometimes occurred when fetching a DNS record entry.
Analytics: Removed legacy Piwik references
Registration Process: Moved on to a dedicated, standalone page rather than trying to drive it through a modal window / iframe.
Registration Process: Deleted legacy registration files and references.
Preferences: Fixed the notification settings options i.e. they don't error now when you try to change their values.
Update 24/11/17
UI: Fixed width issue with embedded charts.
Update 20/11/17
Back-end: Upgraded to PHP 7.1.
Back-end: Updated CMS detection library.
Crawl: Added support for HTTP Authentication credential passing.
Admin: Added the site to the Yandex and Bing search consoles.
Dashboard: Improved the Invalid URI report, with a URI suggestion column to remedy the issue.
Download: Log cleanup.
Ajax Functions: Log cleanup.
Content: Updated content on Features page with sunburst visualisation.
Update 19/11/17
JS: Upgraded to HighCharts 6.0.3
Update 18/11/17
Back-end: Fixed date bug introduced through upgrade to PHP7.1.
UI: Fixed various mobile CSS issues and problems introduced as a result of browser zooming.
Update 15/11/17
Dashboard: Added a new sunburst visualisation type.
Dashboard: Generated several new back-end reports and data structures to facilitate the sunburst chart type.
Update 01/11/17
JS: Upgraded to HighCharts 6.0.2
Update 31/10/17
Dashboard: Added a new treemap visualisation type.
Update 29/10/17
Dashboard: Created n new reports which analyses the terms at each URI directory level depth
Dashboard: Created n new reports which analyses the incremental terms from the root URI
Update 27/10/17
Homepage: Fixed search responsive design
Update 22/10/17
Back-end: Cleaned up various log messages that occurred due to failure to connect to external service
Update 18/10/17
Back-end: Massively optimised the initial database creation. Now happens almost instantly, as opposed to taking around 20 seconds.
Back-end: Improved performance of parsed page storage.
Back-end: Removed DB indexes that were not necessary
Dashboard: Now with Pixel lengths of page titles of how they are rendered in the Google Search Results
Update 05/10/17
UX: More user feedback / progress messages during an audit
Back-end: Refactored some classes to use a slightly more modern approach
Update 28/09/17
Bug: Fixed issue where the link to the report becomes available before the file has fully being written to disk. This could result in broken reports, or generated reports not showing at all.
Update 26/09/17
UX: User experience tweaks for logged in users
Update 25/09/17
Global: Introduced meta noindex and nofollow meta tags on application pages that should not be indexed. Also added rel="nofollow" to links that go to these pages.
Update 21/09/17
Homepage: Tweaked the homepage so that the anonymous user search box has more prominence
Infrastructure: Added AWS ElastiCache Memcached node to the stack to store PHP session data
Update 20/09/17
Dashboard: Re-added the colour analysis back to the Developer Helper report
Update 19/09/17
More log message notice and warning clearing up
UI: Added Apple Touch Icons
Update 17/09/17
Log message notice and warning clear up
Update 15/09/17
Dashboard UI: Provided new technique for condensing very wide tables. Can be seen on the 'Full Link Data' and 'All Response Headers' report.
Crawl: Created data schema to support Dublin Core tags
Crawl: Extended meta tag extraction functionality to be able to parse and store all Dublin Core tags
Dashboard: New Dublin Core Section created
Dashboard: New Dublin Core Report for - Dublin Core All
Update 14/09/17
Dashboard: New Open Graph Tag URL Mismatches report
Dashboard: New Canonical <link> Mismatches report
Dashboard UI: Improved the readability of reports by automatically hiding columns where there is no data within the column
Update 13/09/17
Bug: Fixed bug where the app could crash when a page redirects into an asset.
Status: Added information regarding the current page that the crawl is on.
Crawl: Created data schema to support Twitter Cards
Crawl: Extended meta tag extraction functionality to be able to parse and store all Twitter Cards
Dashboard: New Twitter Card Section created
Dashboard: New Twitter Card Report for - Twitter Card All
Dashboard: New Twitter Card Report for - Twitter Card Sites
Dashboard: New Twitter Card Report for - Twitter Card 'Creators'
Dashboard: New Twitter Card Report for - Twitter Card Types
Dashboard: New Twitter Card Report for - Twitter Card Player
Dashboard: New Twitter Card Report for - Twitter Card App iPhone
Dashboard: New Twitter Card Report for - Twitter Card App iPad
Dashboard: New Twitter Card Report for - Twitter Card App GooglePlay
Bug: IP detection now works. Some crawls would hang previously upon initiation.
Update 23/08/17
Bug: Fixed anonymous audits
Crawl: Improved Debug screen
Status: More user statistics and insight
Dashboard: Added new descriptions to some sections
Update 14/08/17
Front-end: Added a theme-color meta tag to enhance mobile experience <meta name="theme-color" content="#333333" />
Dashboard: Added 10 more newly detected HTTP headers
Security: Added 'X-Content-Type-Options: nosniff' header to improve security.
Update 12/08/17
Security: Added 'Subresource Integrity' checks to external scripts and stylesheets i.e. those loaded from a CDN.
Security: Added 'Secure' flag to cookies set via PHP
Security: Ensured cookies can only load over HTTPS
Security: Set 'X-Frame-Options: SAMEORIGIN' to prevent the site being loaded in an iframe
Security: Added header "strict-transport-security: max-age=63072000" to enhance HTTPS security
Security: Added header "X-XSS-Protection: 1; mode=block" to help prevent XSS
Security: Added header "Referrer-Policy: strict-origin-when-cross-origin" to improve privacy
Security: Added a "Content-Security-Policy" header, albeit a weak one
Update 08/08/17
Status: Added statistics around queue state
Status: Added more progress update messages
Update 04/08/17
Setup: Fix for showing hyphens in the progress update status
Setup: UI fix for the transition between the crawl and the report generation progress indicators
Status: Added a site maintenance mode toggle
Back-end: Created new library file for report generation specific actions
Registration: Fixed bug with registering new users
Update 03/08/17
Sysadmin: Completed migration to new infrastructure
Sysadmin: Applied CloudFront CDN
Sysadmin: Re-pointed all DNS, and decommissioned old infrastructure
Update 02/08/17
Crawl: Queue system in place so that only 2 reports can be generated concurrently
Back-end: Basic system to manage the report generation queue
Sysadmin: Migration scripts to move the application to a new improved stack
Sysadmin: Dedicated NFS to store all crawl and generated report data
Back-end: Various application changes to enable the migration from Apache web server to Nginx
Sysadmin: Upgrade to Ubuntu 16.04 and nginx
Dashboard: Improvements to the way cached reports are detected and called
Install: Install script now covers infrastructure dependencies
Update 20/07/17
Dashboard: Improved UI so that it better displays long unbroken strings, such as those found in cookies.
Install: Created install script to import GeoIP databases.
Back-end: Minor code updates to clear up log messages introduced as a result of the upgrade to PHP7 and MySQL 5.7
Dashboard: Fixed Bootstrap mark-up bug introduced when panes are re-organised.
Update 10/07/17
Status: User stats on a per day basis.
Update 06/07/17
Back-end: Improved SQL queries throughout so that they are compatible with MySQL 5.7.
Back-end: Improved PHP throughout so that it is compatible with PHP 7.0.20.
Back-end: Improved code for the dedicated 'dev' environment configuration.
Back-end: Improved install process.
Update 27/06/17
Dashboard: Added new report for analysis of page Content within the <body>.
Update 22/06/17
Setup: Slightly improved labels and descriptions.
Dashboard: Added new report that lists all discovered Google Analytics tracking urls.
Dashboard: Re-jigged the Left Hand Navigation, and tweaked layout on the overview screen.
Dashboard: Multiple issues fixed with the way Bootstrap was being used.
Crawl: Created data structure to capture sentiment analysis.
Back-end: Integrated Sentiment Analysis library.
Dashboard: Now analysing all title and meta description tags with the sentiment analysis functionality.
Dashboard: New reporting section for Sentiment Analysis with includes analysis of title and meta description tags.
Back-end: Extended the Sentiment Analysis data schema to support Heading Elements
Dashboard: Added sentiment analysis reports for heading types 1-6.
Update 21/06/17
Crawl: Added an exclusion list of external domains that have bot blocking measures in place. This includes and Simple workarounds are possible by using different User-Agent, but not implementing.
Dashboard: Created a new report for viewing robots.txt contents. This also now includes User-Agent information.
Dashboard: Extend the DNS information report to include the root domain record.
My Reports: Improved UI for users who haven't run any reports.
Dashboard: Report that lists all URLs that contain parameters, and another that shows all those without.
Update 20/06/17
Crawl: Fixed bug in parsing the HTTP headers, when the header contained a ':' such as part of a url.
Dashboard: Fixed bug on the left hand menu that prevented page anchors working on certain menu options.
Dashboard: Improved loading efficiency of single report view.
Dashboard: New report that show analysis on the x_drupal_cache HTTP Header.
Dashboard: New report that show analysis on the x_cache HTTP Header.
Dashboard: New report that show analysis on the cache_control HTTP Header.
Dashboard: Improved UI so compress the report grid, in cases where you get several reports that are stacked.
Dashboard: Made UI more consistent by adding the report nav buttons (CSV ) across more reports
Dashboard: Fixed bug with the menu generation of previous audits.
Dashboard: Improved UI of single report screen.
Crawl: Created data structure for parsed robots.txt data
Crawl: Capturing robots.txt data
Update 19/06/17
Dashboard: Added share buttons on the single report view.
Dashboard: New report showing the counts of each HTTP response header type.
Dashboard: Added descriptions about common HTTP response headers (content taken from WikiPedia).
Update 18/06/17
Upgraded Jquery to 3.2.1
Update 31/05/17
Dashboard: Added new 'share' link that allows users to share / view a URL with a single report. Handy for sharing with clients.
Update 13/03/17
GeoIP: Surfaced GeoIP output to show the geographic location of where the website resides.
Back-end: Cleared up some log message notice's and warnings.
Update 12/03/17
GeoIP: Built GeoIP import scripts.
Status: Fixed SQL errors on reporting.
Update 09/03/17
Markup: Added preconnect tags to the document head.
Setup: Added updated set of user agents.
Update 04/03/17
Status: Updated stats for 2017
Update 24/02/17
Dashboard: Can now detect what CMS is being used, from 15 of the most common systems.
Images: Added an SVG logo library to the project, containing flags for every country and region across the globe, which can be used to enhance the UI.
Images: Added an SVG country flag library to the project, containing approx 1000 logos, which can be used to enhance the UI.
Email: Notification emails now include a small thumbnail screenshot of the website being audited.
Dashboard: Added links that are discovered in <link> tags to the broken link checker report.
Update 19/02/17
Dashboard: Added website improvement suggestions to the overview and score pages.
Update 14/02/17
Email: Recommendations added to email
Score: Improved data structure to handle scores
Update 13/02/17
Dashboard: It is now possible to load the next 10 result in any given report, rather than download the whole report.
Score: Added new data structure to support explanations of how the scores are deduced.
Score: Added loads more detail to the score page so that users can see where they need to improve aspects of the site.
Score: Factored broken image src's into the score
Update 06/02/17
Security: Tightened up access to parts of the application.
Preferences: Added extra protection around options such as "Delete My Account".
Preferences: Now shows what membership tier a user is on. This is also visible in the top nav.
Payments: Integrated the application with Stripe to manage payments and subscriptions.
Plan page: Create a pricing / plan page.
Update 03/02/17
Homepage: Improved responsiveness.
Update 28/01/17
Dashboard: Added an area line-chart type for illustrating things such as pages at depth.
Dashboard: Added support for customisable axis label on graphs via the rel attribute.
Update 26/01/17
Homepage: Fixed some responsive issues so the navigation works better on mobile and tablet devices.
Dashboard: Graph to show distribution of image types.
Dashboard: Graph to show pages at a given depth.
Dashboard: Graph to show content types of pages.
Dashboard: Misc graph additions throughout.
Update 25/01/17
Crawl: Now capturing the full HTTP header response for every webpage.
Crawl: Storing full HTTP header response in a dynamic structured format that allows for integration into existing reports.
Dashboard: Made first version of the HTTP header responses visible on the report.
Update 08/01/17
Dashboard: Fixed HTTP analysis links from 'Issues' area into the actual reports.
Update 04/01/17
Crawl: Added better support for discovered compressed and executable files.
Emails: Bug fix for email notifications.
Update 28/12/16
Bug: Fixed issue where a site homepage could sometimes be excluded from an audit. Thanks to Jesse Skeens for reporting.
Bug: Improved favicon detection
Dashboard: Improved UI, so that empty columns are hidden on certain tables.
Update 26/12/16
Bug: Better handling of internal URIs that start with //
Dashboard: New report for images - showing images by type e.g. 100 image/jpeg, 50 image/gif etc.
Dashboard: New report for images - showing favicons by type e.g. 100 image/, 100 image/png etc.
Dashboard: New report for images - showing unique favicons
Update 22/12/16
Crawl: Fixed bug with how paths beginning with // were resolving.
Dashboard: New report showing total weight of assets (images, scripts and stylesheets), by page.
Dashboard: Added the image, style and stylesheet size metrics to the performance area.
Dashboard: Removed the 'misc data points' from the Overview section.
Update 21/12/16
Crawl: Now capturing metrics on responses of Script and Stylesheet assets. Also the original SRC and HREF values.
Dashboard: New report on Broken Script SRC's.
Dashboard: New report on Total Weight of Scripts, by page.
Dashboard: New report on Largest Single Scripts.
Dashboard: New report on Broken Stylesheet HREF's.
Dashboard: New report on Total Weight of Stylesheets, by page.
Dashboard: New report on Largest Single Stylesheets.
Update 13/12/16
Email: Much improved automated email notifications and messaging.
Email: Created HTML email template.
Email: Cleared out legacy Mandrill code, in favour of AWS SES.
Responsive CSS: Improvements made throughout.
Update 11/12/16
Crawl: Now capturing the original image SRC attribute, and the resolved path.
Dashboard: Cleaned up handling of reporting of images that use a data: type within an image SRC.
Dashboard: New report to show images that use a data: type within an image SRC
Dashboard: Extended image reporting to show example page urls of pages that have image problems.
Dashboard: Added new report to show the total weight of images, per page URL.
Dashboard: Fixed image report that shows pages that have images without any alt text set.
Email: Improved the emails that the tool auto-generates, by making them HTML responsive.
Update 28/11/16
Dashboard: New report to show linked asset sizes summary, so that the raw HTML page weight summary can be seen separately.
Dashboard: Further enhanced the Image reporting, with example source URLs
Dashboard: New report to show number of pages per language type discovered
Dashboard: Fixed UI for when investigating what pages a spelling mistake is on.
Update 27/11/16
Contact: Fixed the form. Dodgy logic was preventing the contact messages coming through.
Upgraded Jquery to 3.1.1
Dashboard: Next to each URL, there is now a button that allows you to inspect the HTTP Headers.
Dashboard: WHO IS lookup now works for HTTPS protocol. This was a bug fix.
Dashboard: Updated and added various copy on the reporting screen.
Setup: Added additional validation and user feedback for domains which can't be crawled.
Back-end: Cleaned up code to prevent a variety of log messages being recorded for PHP notices and warnings.
Update 21/11/16
Crawl: Now capturing the full HTTP response from image files
Dashboard: Showing full HTTP response for image files
Dashboard: Showing largest images discovered on the site.
Update 17/11/16
Crawl: Fixed bug around restricting audits to a given language. The 'all' option was behaving inconsistently.
Update 16/11/16
Back-end: Improved debugging information.
Update 30/9/16
Theme: Changed the default theme to something less default BootStrap based.
Content: Tweaked copy throughout.
Menu: Added 'Features' Menu item.
Update 20/9/16
Dashboard: Invalid internal URI's are now tested for
Update 12/9/16
Dashboard: Fixed the internal link references for the duplicate content reports i.e. it now correctly lists all pages that are discovered with the same duplicate content
Dashboard: Fixed the Canonicalised domain check, by performing deeper iterative redirect checks
Dashboard: Added report to show all Open Graph tags
Dashboard: Now collecting and showing hreflang tags on the audit reports
Dashboard: Now collecting and showing DNS Prefetch tags on the audit reports
Update 03/9/16
Dashboard: Added metrics for largest and smallest page sizes that have discovered
Update 02/9/16
Thumbnail generation throughout is now automatic if cached versions are not detected
Update 27/8/16
Dashboard: Fixed bug with generating the website screenshots. Screenshots are now also auto-trimmed to get rid of surplus white space.
Dashboard: Generated full page image screenshot of the target page which can be downloaded by users.
Email: Check to ensure users are verified before sending audit report emails.
Update 26/8/16
Dashboard: Added fastest and slowest page performance metrics.
Dashboard: Added shortest and longest page metrics.
Update 25/8/16
Added more usage statistics to the administration usage page.
Update 24/8/16
Upgraded various JQuery plugins, DataTables, and tweaked scripts to support JQuery 3.
Update 16/8/16
Upgraded Bootstrap to 3.3.7
Upgraded JQuery to 3.1
Upgraded Highcharts to 4.2.6
Update 16/7/16
Crawl: Audits can now be restricted to any given language, providing that the html tag has the lang attribute set e.g. <html lang="en">
Back-end: Now capturing information about the pages language which can be used in analysis
Bug: Fixed the contact form
Update 24/6/16
Back-end: Improved capture of scores, so that analysis is more efficient.
Dashboard: New report that captures redirections upon entry to a site.
Dashboard: Improved rendering rules of data points, and forced 404 output.
Score: Factored in the number of redirections upon entry to a site.
Score: Factored in the how the site handles forced 404 errors.
Setup: Added a new minimum debug focused crawl that can be run with a single click.
Crawl: Can now capture variables at any point during the crawl, and report generation process.
Update 8/6/16
Back-end: Improved debugging of crawls where there is a lot of redirection upon initially hitting the domain.
Update 4/5/16
Back-end: Added email test facility
Update 3/5/16
Back-end: Now sending emails through Amazon SES
Install: Process for installing the application for the first time is improved, with more feature detection and better messaging.
Update 25/4/16
Status: Optimised administration reporting queries. Reporting section now loads 75% quicker.
Update 21/3/16
Setup: Can now configure if links discovered in frames, iframes and image map area tags should be crawled.
Score: 'F' grade score colour fixed.
Update 15/3/16
Crawl: Major bug fix on multi-threaded crawls that resulted in partially downloaded pages.
Setup: Added crawl depth feedback so that users can better understand the progress of a crawl.
Update 14/3/16
Back-end: Improved ability to cancel crawls.
Back-end: Improved how, and the frequency in which data is saved to disc.
Back-end: Improved how audit progress tables are saved to disc.
Report: Fixed report that shows the Queued links.
Report: Fixed issue with reporting information of site error handling and favicon presence for target urls entered without a trailing /.
Crawl: Fixed bug that caused the crawler to ignore nofollow instructions on multi-threaded crawls
Crawl: Fixed bug that caused links to sometimes be omitted from a multi-threaded crawl
Update 12/3/16
Back-end: Code clean-up and commenting in reporting library.
Content: Polished the copy throughout the information pages.
Setup: Admin helpers to do a quick full crawl of a site.
Update 11/3/16
Crawl: Fixed detection of mailto: and javascript: where case wasn't lower.
Status: Added source user IP analysis
Status: Added source user join date analysis
Status: Added source user last login date analysis
Update 2/3/16
Setup: Fixed issue where something the report wouldn't show, if you closed the browser windows during the audit.
Update 29/2/16
Back-end: Integrated xhprof into the application for improved performance analysis when developing locally.
Dashboard: Fixed caching issue for when you force the regeneration of a thumbnail
Status: Added more application statistics
Status: Improved the users contacts UI
Update 24/2/16
My reports: Improved UI.
Update 23/2/16
Dashboard: Improved UI rendering of tooltips.
Dashboard: Improved UI rendering of linking pages (when you click the icon).
Front-end: Upgraded to jQuery 2.2.1
Dashboard: Fixing 'copy' functionality and added fallback support from browsers that do not have Flash installed / enabled
Update 20/2/16
Admin: Improved application install process.
Update 19/2/16
Crawl: Fixed bug relating to capture of email address using the mailto: protocol in a href. Instances of this where it contained a ? resulted in an error.
Dashboard: Fixed bug relating to direct links into # tab anchors
Dashboard: Upgraded HighCharts to 4.2.3
Homepage: Added carousel block with more information around what the tool can do
Update 18/2/16
Dashboard: The "My Reports" and "Showcase Reports" dropdowns are now alphabetical, and unrestricted in length.
Crawl: Audits can now be run with the shortest urls analysed first
Crawl: Audits can now be run with the longest urls analysed first
Front-end: Upgraded to jQuery 2.2.0
Front-end: Upgraded to DataTables to 1.10.11
Back-end: Upgraded to QueryPath to 3.0.4
Update 12/2/16
Upgraded to PhantomJS 2.1
Dashboard: Fixed issue with certain reports missing download links
Dashboard: Fixed notice in CSV downloads alerting of 'undefined variable'
Update 10/2/16
Fixed issue with latest version of CURL that prevented gzipped HTML being read properly.
Update 3/1/16
File system: Fixed all file naming conventions that aren't Windows compatible.
Update 2/1/16
Crawl: Prevented the crawling of certain types of url extensions.
Crawl: Prevented the crawling of certain domains.
Update 21/12/15
Crawl: Improved crawl failure messaging.
Status: Banning by activity, based on IP is active.
Status: Can add IP's to a ban list through the admin UI for either crawls, messages or both.
Status: Implemented the DB structure for the IP banning functionality to counter spam crawls and messaging.
Update 4/12/15
Status: Optimised report table loading.
Update 30/11/15
Back-end: Added new dictionary database table containing all English swear words.
My Reports: Fixed bugs relating to incorrect parameter being used on the dashboard links.
Crawl: Can now crawl pages that return a 303 HTTP code.
Dashboard: New Content Report - Scans website text for any occurrences of "lorem ipsum" copy.
Update 26/11/15
Back-end: Cleanup of library file structure.
Upgraded Bootstrap to 3.3.6
Update 05/11/15
Back-end: Can now manipulate global application settings via the admin UI.
Reports: Improved mark-up so that it makes better use of Bootstrap defaults
Reports: Fixed bug on spelling pane when bringing in full content
Reports: Added a case toggle on the spelling pane
Update 04/11/15
Back-end: Added global settings table to allow for better application configuration management.
Back-end: Began migrating hard-coded settings to settings dns_get_recordtable to allow for manipulation via admin UI.
Update 08/10/15
Back-end: DataTables can now load in data from the server.
Back-end: Migrated all usages of DataTables to load data from the server, as opposed to pre-loading it on page load.
Update 07/10/15
Back-end: Set up new SQL queries to allow DataTables to load additional data via AJAX.
Update 25/09/15
Dashboard: Update HighCharts library to 4.1.8.
Update 23/09/15
Status: Integration of Google Analytics reports into the administration view.
Update 22/09/15
Status: Better feedback for administration tasks i.e. number of audits affected.
Update 14/09/15
Forgot Password: Improved validation.
Update 08/09/15
My Reports: Now with option to delete old reports.
Dashboard: Removed needless visualisation value in misc data points.
Dashboard: Changed misc data points to be a table.
Dashboard: Links from the overview section, now go to the right "pane" and right report.
Update 01/09/15
Dashboard: Updated copy.
Information pages: Updated copy with some of the newer reports and features, such as spell checking.
Content: Turned the tool on ourselves and fixed a load of spelling mistakes!
Updated DataTables to 1.10.9
Server: Overhauled the .htaccess file to contain latest updates and best practices.
Update 29/08/15
Dashboard spelling fix: prevent concatenation of words
Update 26/08/15
Crawl: Custom dictionary for spell checking
Dashboard: Surfaced first basic report for spell checking
Dashboard: Functionality to show which pages spelling mistakes occur on
Dashboard: Words that potentially contain spelling mistakes can be looked-up on an online dictionary
Update 25/08/15
Dashboard: Fixed bug where same domain links, on a different protocol were being treated as external links.
Score: Integrated no HTTPS in to the score
Dashboard: Fixed bug with generation of the full link data download of a site
Dashboard: New report that lists internal absolute URLs
Dashboard: New report that lists internal relative URLs
Dashboard: Removed the Social and SEO reports because they aren't really relevant to the tool
Dashboard: Added support for Google Tag Manager in the Analytics detection
Crawl: Added pspell library in preparation for spell checking functionality
Crawl: Back-end for spell checking, and suggestion functionality
Crawl: DB structure to handle spell checking
Update 24/08/15
Dashboard: Fixed domain canonicalisation bug
Dashboard: Improved UI with a close button on the fixed LHN
Dashboard: Tests if a site has a HTTPS version or not
Update 06/07/15
Upgraded Bootstrap to 3.3.5
Dashboard: Fixed queued links report.
Audit email notification: Added some high-level audit result information.
Update 12/06/15
Dashboard: Added report for broken image SRC's
Dashboard: Added report for HTTP Status codes of external sites and domains
Score: Factored non-200 http status code external links
Update 19/05/15
Improved admin status screen with minor UI tweaks
Email alerts for 'admin' users when new contact messages are received.
Main Navigation tweaks
Update 10/05/15
Updated jQuery to 1.11.3
Update 25/04/15
Back-end: Code cleanup after running static analysis tool over code base.
Update 21/04/15
Users: Created an upgraded standard user account role type that can crawl up to 5000 urls.
Setup: Updated user agent settings.
Setup: Added option to use a proxy service for pro users.
Back-end: Cleanup of obsolete jQuery and Bootstrap library files.
Updated DataTables to 1.10.6
Update 29/03/15
Setup: New option to force ASCII crawl mode.
Setup: Detect if a site is behind a http auth: 401 error code, and alert user. Also handles other http status codes that prevent an audit from happening.
Update 27/03/15
Bug fix: Set the initial referrer to as some sites were rejecting the request with a HTTP 400 code.
Update 21/03/15
Upgraded Bootstrap to 3.3.4
Update 17/03/15
Dashboard: New visualisation - External links, by domain, with and without social networks.
Update 13/03/15
Dashboard: Fixed bug with visualisations.
Score: Factored in multiple occurrences / types of web analytics code into the score calculation.
Update 12/03/15
Content: Updated the Recommended SEO Tools page.
Update 01/03/15
Setup: UI tweaks, along with the introduction of a "Tour" feature.
Update 26/02/15
Crawl: Increased the speed of the crawling process by a factor of 10!
Update 20/02/15
Dashboard: New downloadable sitemap.xml files.
Update 19/02/15
Dashboard: New downloadable CSV report for all internally linked pages.
Dashboard: New report for Page Rank of internal pages.
Update 18/02/15
Dashboard: Added "copy" feature to "Pages Linking To" feature
Update 17/02/15
Dashboard: Fixed bug relating to "Pages Linking To" feature
Update 30/01/15
Dashboard: New reports for documents and media assets
Update 29/01/15
Fix crawl bug that wasn't handling URL queries properly
Update 26/01/15
Dashboard: New Developer Helper - Download URLs of each HTTP type as an Array
Update 19/01/15
Upgraded Bootstrap to 3.3.2
Update 04/01/15
Dashboard: New report for viewing canonical <link>
Dashboard: New report for viewing shortlink <link>
Dashboard: New report that shows the number of pages at each depth level within the site
Update 03/01/15
Audit settings: Anonymous users now get a 25 page audit, and registered users get a 500 page audit. Copy updated to reflect this.
Dashboard: HTML markup improvements
Update 02/01/15
Email: Added email notifications to platform that let users know once an audit it complete.
Email: Improved email content.
Email: Added DKIM and SPF records + verified the domain. Auto-generated emails now come from
Setup: Improved the UX to let users know what the application is actually doing, rather than just asking them to be patient. This is important for larger audits.
Platform: Separated the Report generation for the report viewing.
Update 01/01/15
Dashboard: 404 Error Response code detection report for directories, html files, and scripts.
Update 31/12/14
Platform: Full site wide HTTPS now integrated
Dashboard UI: Addition of single click button, to scroll to top of page.
Update 30/12/14
Dashboard: Added RSS feed detection and reporting to the 'Content' section
Update 29/12/14
Crawl: Identified a major problem with how relative URL's are resolved, and fixed it. How embarrassing.
Homepage: Better resizing of the page when using different themes
Update 27/12/14
Crawl: Can now extract links from <area>, <iframe> and <frame> tags
Update 26/12/14
UI: Improved UI of the preferences page
UI: Added functionality that allows users to choose their own UI theme on the preferences page
Update 23/12/14
Updated jQuery to 1.11.2
Update 10/12/14
Updated DataTables to 1.10.4
Dashboard: Zoomable screenshot thumbnails, along with optimised resolutions for them.
Update 9/12/14
Dashboard: New report that shows internal links with invalid URL's, specifically those with more than one question mark.
Dashboard: Added tooltip helpers to the various icons used throughout
Update 7/12/14
Crawl: Can now handle stylesheets in this format: <style>@import url("stylesheet.css");</style>
Update 6/12/14
Dashboard: Fixed bug on the website thumbnail generation whereby homepage redirects were not handled properly.
Dashboard: Upgraded phantomjs to 1.9.8
Dashboard: Added a radar chart for the score visualisation
Crawl: Fixed bug relating to the crawl not following 301's properly
UI: Content markup tweaks to take advantage of some of the newer Bootstrap features
UI: Added scores to the report examples
Update 29/11/14
Crawl: Fixed bug that caused CDN asset URI's that contain '//' at the start to be reported as an internal link.
Update 21/11/14
Upgraded Bootstrap to 3.3.1
Update 25/10/14
Homepage: Content tweaks and responsive enhancements
Update 24/10/14
Crawl Options: Whether or not cookies should be accepted on a crawl can now be toggled.
Update 21/10/14
Users can now cancel an audit from the setup screen.
Update 15/10/14
Set the production site to use a CDN for Bootstrap and jQuery.
Update 09/10/14
Content UI: Added breadcrumbs to content pages.
Update 05/10/14
Dashboard: Added new developer helper that shows the RGB and HEX codes of the most prominent colours on the website homepage.
Update 24/09/14
Server: Upgraded to a more performant server.
UI: Improved the responsive behaviour for tablet devices.
Dashboard: Fixed 'Link Types' visualisation bug.
Update 19/09/14
Setup: Better handling of the URL on the setup screen. Thanks to Jack21222 for reporting it.
Update 15/09/14
Content: Copy tweaks throughout.
Dashboard: Overview screen tweaks.
Dashboard: Added developer helper to help fix any Domain Canonicalisation issue.
Update 11/09/14
Dashboard Bug: Fixed visualisation.
Update 09/09/14
Homepage: Main Banner Height Fix
Update 07/09/14
Dashboard: Refined output of audit overview screen
Score: Factored in Domain Canonicalization
Update 06/09/14
Dashboard: Created new audit overview screen
Score: Captured additional information used in score generation
Update 04/09/14
Dashboard: Added recrawl button to admin navigation
Update 02/09/14
Registration: Users are now automatically logged-in after completing the registration process.
Crawl: Previous audits can now be re-run on demand
Crawl: Proxy service is only now used for external urls
Update 30/08/14
Crawl: Fixed bug to do with saving the crawl every 50 pages.
Crawl: Capturing status code information about external links.
Update 29/08/14
Setup: Can now choose whether to check external links to see if they work
Crawl: Checks external links, then as a fallback, checks the domain
Dashboard: New report on broken external links
Dashboard: New report on broken external domains
Update 25/08/14
Crawl: Fixed bug that sometimes occurred for anonymous crawls
Setup: Improved anonymous user journey
Update 24/08/14
Status: Better error flagging and a few misc. bugs.
Dashboard: Now reporting on the robots.txt crawl delay
Crawl: Fixed bug that prevented single page sites producing a valid report.
Crawl: Fixed ability to cancel crawls
Setup: Estimated time remaining for an audit is now shown on the progress modal window
Update 20/08/14
Dashboard: Added Report for captured strong / em / b / i tags
Dashboard: Added Report for captured object / svg / canvas tags
Dashboard: Added Report for Deprecated tags inc. applet / basefont / center / dir / font / isindex / menu / s / strike / u
Update 19/08/14
Dashboard: Favicons next to the showcase links
Update 18/08/14
Back-end: Cleaned up logs
Update 15/08/14
Dashboard: Improved UI to only show reports that are available.
Content: Updated the FAQ
Dashboard: When toggling between different versions of audits, the pane will remain the same for each version.
Update 14/08/14
Dashboard: Can now load previous versions of an audit
Dashboard: Misc. minor bugs thrown up in the logs
Back-end: Restructured how audit report information is stored.
Crawl: Crawl data is now saved to disc every 100 pages. This lay the foundations for pausing crawls, and restarting crawls after an error.
Back-end: Error reporting toned down.
Update 06/08/14
Crawl: Set default crawl order to be the most frequently occurring links first. This will make the page limited reports more useful.
Dashboard: Added report to show the largest images, by surface area.
Dashboard: Added report to show images with invalid values in the width and height attributes.
Performance: Added performance monitoring and metric capture to help identify performance bottlenecks.
Performance: Initial generation of report is now ~50% quicker.
Dashboard UI: Any reports that don't have any data, are hidden.
Update 05/08/14
Dashboard: Improved UI of Audit Overview page
Dashboard: Added 'Full Link Analysis' Report that shows information on URL path, query and fragment.
Update 04/08/14
Content: Added base table structure for the glossary of terms, and the SEO tool comparison analysis.
Dashboard: Added new section called 'Developer Helpers'
Dashboard: Added new 'Developer Helper' called '.htaccess Redirects'
Dashboard: Added new 'Developer Helper' called 'VHOST Redirects'
Dashboard: Added detection for deprecated HTML tags.
Dashboard Bug: Fixed domain canonicalisation reporting bug.
Update 03/08/14
Crawl: Now capturing strong, em, b, i, object, svg, canvas tags. No analysis on them yet though.
Crawl: Added capture of Google Analytics information
Dashboard Bug: Fixed bug that sometimes occurred when clicking the which resulted in no results being returned.
Dashboard: Added analytics information on Audit Overview page
Dashboard: Added new report 'Occurrences of each analytics / tracking scripts'
Crawl: Added lookup tables
Content: Updated the About Us page content.
Update 02/08/14
Crawl Bug: URLs that were entered as a start point, that then redirected to different sub-domain / domains wasn't working properly. Now it is.
Update 01/08/14
Dashboard Bug: Fixed bug that resulted in inaccurate page speed and size values being reported due to inclusion of non-status code 200 pages.
Dashboard: Added additional helper copy.
Dashboard: Added "Redirect Speed" report.
Dashboard: Tweaked the layout on the Performance reports.
Dashboard: Added anonymous audit user sign-up encouragement.
Dashboard: Colour coding on the tables to signify scale of the issue.
Crawl: Created facility to capture extra html tags for analysis.
Update 26/07/14
Setup: Added more information to the "in progress" modal window so that users know where in the process the audit is.
Dashboard: Admin Navigation - Previous and next links to cycle through the dashboards. Plan is to make this available to users.
Dashboard: Admin Navigation - Make an audit public.
Dashboard: Added loading animation when users click the 'see more' links.
Update 23/07/14
Content: Updated the Privacy Policy with cookies information, and added the TOS.
Update 20/07/14
Leaderboard: Capture score data for building leaderboard
Leaderboard: Initial version of the Leaderboard page
Setup: Added additional information to the modal window
Update 19/07/14
Leaderboard: Creation of base tables to support a leadboard
Update 18/07/14
Dashboard: Bug fixed that was introduced as a result of upgrading to the latest version of Bootstrap relating to the progress bar.
Update 13/07/14
Crawl: Captured additional attributes from <img> tags.
Dashboard: Added image information to the report.
Dashboard: Added unique image report.
Dashboard: Added most common image report.
Dashboard: Added image that don't have any 'alt' text set report.
Dashboard: Added images with 'title' attribute set report.
Update 12/07/14
Dashboard: Added extra error detection for broken crawls.
Dashboard: Started adding the image analysis.
Update 08/07/14
Upgraded Bootstrap from 3.1.1 to 3.2.0
Update 06/07/14
Dashboard UI: Added a pre-page loader, so that when large reports are loading, users know what is going on.
Update 04/07/14
Markup: Added rel="nofollow" to all of the login / registration links on the site, and excluded them from robots.txt to prevent issues with some bots.
Status: Anon crawls can now be delete en-mass
Tracking: Pane clicks on the dashboard and now captured in Piwik
Update 29/06/14
Status: Cleaned up admin options menu.
Update 27/06/14
Added server monitoring software.
Update 23/06/14
Crawls now go via a proxy service.
Update 21/06/14
Started improving, and enforcing a structure on the JS
Update 21/06/14
Performance: Reduced the page size of the dashboard by dynamically calling the keyword data.
Dashboard: Made the site URL clickable, and sorted out the CSS index issues on the pane headers.
Update 18/06/14
Content: Published a Recommended SEO Tools page.
Update 14/06/14
Dashboard: Links to the dashboard now support passing the url via the query e.g.
Dashboard: Better handling of errors for dashboard loading and retrieval
Homepage: Added public crawl screenshots to the site homepage, and tweaked the layout
Bug fix: Removed debug code from the sign-up process
Status: Take DB back-ups
Content: Added a report example sitemap page
Dashboard: Added "full screen" links next to all of the reports
Update 08/06/14
Crawl: Added function to count the number of words on a page
Crawl: Added function to count the number of sentence's on a page
Crawl: Added function to calculate the reading time of a page
Crawl: Added function to calculate the content Gunning Fog Index of a page
Crawl: Added function to calculate the content Flesch Reading Ease, and the Flesch–Kincaid Grade Level of a page
Crawl: Added function to calculate the content Coleman–Liau Index of a page
Crawl: Added function to calculate the content SMOG grade of a page
Crawl: Added function to calculate the content Automated Readability Index of a page
Crawl: Added function to calculate the content Dale–Chall readability of a page
Crawl: Added function to calculate the content Spache Readability Formula of a page
Dashboard: New report to show all of the content related information.
Update 07/06/14
Crawl: Extended the page details table to support additional content data capture
Update 06/06/14
Content: Added an FAQ section
Bug fix: Multiple values with the anchor 'rel' attribute were being analysed incorrectly.
Bug fix: Sometimes linking pages was not returning any results.
Update 04/06/14
Dashboard: Improved toggles on showing / hiding table data
Update 31/05/14
Setup: Only allow 'analyse sitemap.xml' when selecting a full crawl
Status: Added email domain link to users view
Update 30/05/14
Bug fix: Links starting with 'tel:' and 'skype:' were resolving as a 404
Bug fix: Relative links on sub-directory pages were sometimes resolving incorrectly
Update 28/05/14
Analytics: Upgraded to Google Universal Analytics, and added several click data capture points so that user behaviour can be better understood.
Update 26/05/14
Dashboard: Re-factored keyword analysis code, and extended it to include any number of keyword chains. Currently exposed is: singles, pairs, trios & fours.
Content: Added screenshots to the SEO Analysis page.
Content: Misc. copy tweaks throughout.
Update 24/05/14
Dashboard: Dynamic visualisations of keyword data, just click the next to the report heading.
Dashboard: All visualisations can now be exported in PNG, JPEG, SVG or PDF format.
Homepage: Dynamic testimonials.
Update 22/05/14
Content: Added a Privacy Policy.
Update 20/05/14
Dashboard: New view to report on just keywords
Dashboard: Added report on internal links with rel="nofollow"
Dashboard: Added report on internal links with target="_blank"
Dashboard: Calculated ratio of internal & external follow/nofollow
Dashboard UI: Introduced Pie Charts to the Audit Overview page
Crawl: Upgraded the anchor table to capture whether a path is absolute or relative
Upgraded jQuery to 1.11.1, and Parallax to the latest version.
Dashboard: Download table data that was previously not downloadable as CSV e.g. "Keywords in Anchor Text"
Update 18/05/14
Status: Favicon changes with the progress of the crawl
Dashboard: Integrated HighCharts for the charting / graphing visualisations, after ruling out ChartJS (no label option on pi charts!), Google Charts (reliant on web service), ChartSpree (usage limitations)
Dashboard: Added site audit visualisation functionality.
Dashboard: Added basic visualisation for HTTP Response Status Codes & Link Types
Dashboard: Added basic visualisation for Link Types
Crawl: Upgraded the capacity of various fields, and captured additional link anchor information including the 'rel' and 'target' values
Update 17/05/14
Crawl: Fixed bug that caused bypassing of validation checks on the 301/302 links for 'redirect_url' component
Dashboard: Added a keyword density column on all of the keyword reports
Update 15/05/14
Dashboard: Can now cycle through the different screen resolutions on the audit overview screen
Dashboard: On the duplicates report, you can now see what urls have the duplicate's
Dashboard: Updated the Score UI to include a website thumb, and exciting sliding bar charts!
Setup: Better validation on the url field.
Update 14/05/14
Dashboard Bug: Fixed download reports for the duplicate reports.
Update 13/05/14
Dashboard UI: Show the download and see all links more appropriately
Update 11/05/14
Status: Can now regenerate all public cache dashboards so that they show the latest analysis.
Dashboard: New report - keywords in anchor text
Dashboard: New report - keyword pairs in anchor text
Dashboard: New report - keywords in meta descriptions
Dashboard: New report - keyword pairs in meta descriptions
Dashboard: New report - keywords in meta keywords
Dashboard: New report - keyword pairs in meta keywords
Dashboard UI: LHN sub-navigation is now fixed to make it easier to navigate large reports
Update 10/05/14
Bug: When clicking "see all" on the dashboard, the options disappeared.
Bug: LHN on dashboard. Long title were being truncated on low screen resolutions.
Dashboard: Put the most useful reports to the top of each section.
Update 30/04/14
Bug: 301 redirect detection fixed
Dashboard: Improved 301 / 302 reporting
Update 28/04/14
Dashboard UI: Large tables now scroll horizontally.
Update 27/04/14
Dashboard: New report - keyword pairs in titles.
Dashboard: New report - keyword pairs in headings.
Update 21/04/14
Users: Can now toggle whether or not they want to see the 'tips' on the user preferences page.
Setup: Overhauled the UI on the setup screen to make it easier to use.
Dashboard: Tweaked the UX so that menu contents are better arranged.
Update 20/04/14
Core: Added stop words to the analysis.
Core: Logging system to capture and interrogate information on a per-site basis, rather than centrally.
Back-end: Logging of how long each report takes to generate.
Dashboard: New report - Keywords in headings
Dashboard: New report - Keywords in titles
Dashboard: Can now deep link to top level reports
UI: For Dashboard LHN. Enabled closing menus, and toggle arrow.
UI: Parallax is back involved on the homepage.
Update 19/04/14
Dashboard: Can now regenerate thumbnails
Backend: Integrated Composer to handle dependences
Core: Advanced URL parsing to accurately determine subdomain, TLD domain, and URL suffix.
Dashboard: New report for External suffix analysis
Dashboard: All reports now show 15 results by default, and have "see all" toggles.
Dashboard: Added new tips.
Update 17/04/14
Dashboard: Improved UI of HTTP headers block
Update 16/04/14
Status: Can now clear website screenshot cache
Score: Basic calculations for all areas
Dashboard: External links by domain
Dashboard: External links by domain (follow)
Dashboard: External links by domain (nofollow)
Dashboard: Loading spinner while generating thumbnails, plus UI tweaks for score
Update 13/04/14
Score: Grade letter generation
Score: Dynamic text for score reporting
Dashboard: Shuffle UI to make score the default landing page
Update 06/04/14
Dashboard: Detect links with underscores
Dashboard: Detect links with common language extensions
Dashboard: Calculated URL length metrics
Update 05/04/14
UI: Tweaked mark-up throughout to make better use of Bootstrap capability.
Dashboard: Detect links with spaces
Dashboard: Detect links with bad case
Update 04/04/14
Score: Improved UI for score page, and integrated into Dashboard navigation.
Update 03/04/14
Score: Rendered output of scoring algorithm as bar graph.
Update 01/04/14
Score: Finished algorithm framework for calculating the score.
Update 31/04/14
Score: Finished core work on the website Scoring / Grading
Update 14/03/14
Dashboard: Added result counts to the tables.
Update 13/03/14
Debug & Variables tables created per crawl, to store extra information.
Dashboard: Display of crawl log
Update 11/03/14
Status: Admins can now in-line edit user fields.
Update 24/02/14
Users can now register using a Twitter Account.
Sign-up & login process has been improved, including welcome email.
Dashboard: Content tweaks and additions throughout.
Dashboard: Improved heading analysis.
Update 23/02/14
Dashboard: Added report on queued links.
Homepage: Fixed Parallax issue on Chrome.
Upgraded Bootstrap from 3.1.0 to 3.1.1
Upgraded jQuery to 1.11.0
Update 19/02/14
Dashboard: Added HTTP headers. This means you can now detect the WebServer type, along with Server OS, amongst other things.
Dashboard: Updated layout of audit overview, and improved the rendering of the DNS and WHOIS information.
Dashboard: Added basic Sitemap XML insight.
Dashboard: Increased breadth of WHOIS server list.
Update 13/02/14
Can now restrict crawls by url length.
Can now restrict crawls by directory depth.
Update 11/02/14
Added new data scraping options for extracting content from lists.
Update 09/02/14
Can now deep link to sub-reports on the reports.
Update 04/02/14
Parallax eye candy for the homepage
Update 02/02/14
Upgraded Bootstrap from 3.0.3 to 3.1.0
Added Social sharing buttons to the site footer.
Update 01/02/14
Added Headroom.js to the app.
Update 29/01/14
User sessions can now be preserved with "Remember me" functionality.
Update 26/01/14
Created a logo using the Squarespace Logo tool.
Dashboard: Pages with missing Favicons
Dashboard: Content ratio
Update 12/01/14
Open Graph tags have been added to the dashboard / report page.
Update 09/01/14
CSS and JS consolidation.
Update 05/01/14
Added data scraping info page
Update 04/01/14
Users can now run a 10 page SEO audit without signing up to the application. Click the "try now" button on the homepage under the sign-up box.
Fixed various Chrome browser bugs
Added missing map file for jQuery 1.10.2
Update 03/01/14
Added Google reCAPTCHA to the app, and exposed on the contact form.
Contact form: Remember state on failed submission.
Forgotten password: Fixed bug introduced in upgrading to Twitter Bootstrap 3.0.3
Added a smattering of Glyphicons throughout.
Update 22/12/13
Dashboard: New URL length report.
Dashboard: Bug fix relating to "see all" and download report.
Dashboard: Added screenshots of website homepage taken at different dimensions.
Status screen improvements for cache regeneration.
Update 17/12/13
Small tweaks to admin options / menu
Update 16/12/13
Tweaked wording throughout so it's less "crawl" & "spider", and more "audit" & "fetch".
Upgraded to Twitter Bootstrap 3.0.3
Update 15/12/13
Dashboard: All reports are now downloadable in CSV format.
Update 11/12/13
Dashboard: Added Basic Report containing page title, meta description, meta keywords, first H1.
Setup: Fixed bug relating to the sliders passing the correct values in to the form.
Update 10/12/13
Optimised initial loading time of Dashboard for a new crawl.
Fixed setup screen UI bug for example data selections, and test data crawls.
Initial content for SEO Analysis, Link Analysis, Site Architecture
Update 08/12/13
Dashboard LHN UI bug fix
Added dashboard reports for: All Titles, Missing Titles, All Meta Tags.
Added Glyphicon's to Dashboard LHN
Improved status screen with cache detection, rebuild cache and more user friendly default values
Update 01/12/13
Dashboard page now has a left hand navigation menu
Update 30/11/13
Updated Twitter Bootstrap from 2.3.2 to 3.0.2
Update 27/11/13
Output example now merges data from different crawls
Output example now has a cache lifetime of 1 hour
Added crawl schedule information to the Groupon / Wowcher merge example
Update 23/11/13
Dashboard bug fix for "see all data"
Extended remote triggering of crawls to include by site ID and url
Update 20/11/13
Crawls can now be triggered remotely
Update 17/11/13
Fixed bug that causes the crawl to hang when calculating the content ratio
Sortable fields on the crawl setup
Tests if no data is available for the data scrape example
Upgraded to jQuery v1.10.2
Update 11/11/13
Phased out simple_html_dom as it can't handle pages with huge numbers of links.
Fixed bug due to some DB connections not being closed
Fixed the data view, for scraped data analysis
Fixed Google Analytics reporting + added goal tracking for GA & Piwik
Now have a global exclusion list of urls / domains to be ignored
Update 04/11/13
Status screen options for deleting caches etc.
Update 04/11/13
Each site now has its own cookie file, which fixes a reported shared cookie bug.
Status Screen: Filtering & paging on all_crawls, and the debug messages.
Tightened up the crawling rules around the limits per domain, per day.
Filtering fix as reported on a Reddit thread.
Suppressed error message when a site doesn't have a robots.txt file.
Fixed issue for 'multiple page scrapes' when there is in fact only one page on the site.
Sitemap.xml file added.
Recommendations - basic implementation added.
Dashboard - converted navigation to a mega menu.
Update 03/11/13
Consolidated ajax functions.
Updated Daily Deal Consolidation Example, and added to the site navigation.
Added more user insight on the status screen.
Update 31/10/13
Increased crawl limit for free users to 250.
Added Piwik analytics.
Capture user IP on registration to detect potential spam.
Added About page.
Crawl complete email notifications.
Update 29/10/13
Added forgotten password functionality.
Added more robots.txt rules.
Validation on url field on new crawls.
Update 28/10/13
Beefed up the security around the ajax requests on the status page.
Admin functionality - delete users and newsletter signups.
Registration form - fixed type "email" that some users were reporting failures on.
Update 27/10/13
Added robots.txt
Web Analytics only applied for unknown IP's.
Fixed bug in the HTML parser that was causing some crawls to error out. is_object() detection on str_get_html.
Fixed a couple of bugs reported via Reddit.
Update 26/10/13
Added a Sitemap.
Meta Descriptions & Open Graph tags added to the site.
Newsletter signup.
Update 25/10/13
You can now follow us on Twitter.
Add Google Analytics & Webmaster Tools.
Status page - added user table.
Status page - make public / private toggle.
404 error page.
.htaccess file + crossdomain.xml rules.
Fixed cancelling crawls.
Update 24/10/13
Extended registration process to prevent duplicate user names, and email addresses.
Back button now works when navigating the tabs on the dashboard page.
Crawl progress bar now works properly.
Update 23/10/13
Extended user table to support additional fields.
Implemented user settings / preferences screen.
Added test for unverified and blocked users.
Email validation for new users.
Update 22/10/13
Created user library back-end code.
Added email notifications.
Added close icon to alert notifications.
Dashboard compatibility fix for new PHP version.
Additional server and environment settings detection code.
Update 12/10/13
Added back-end D3 integration.
Update 11/10/13
Updated jQuery library.
Tweaked UI.
Version 1 - 08/10/13