Want to keep up with the latest Firefish Software updates and announcements? You're in the right place!
22nd November - Outlook Add In Login/Email Logging Issue
08:58 - 25th November - Resolved - We have now successfully received approval from Microsoft and deployed a resolution to all users and you should no longer experience further issues with email logging through the Outlook Add-In.
We will continue to investigate this issue to identify the root cause of the issues experienced by users to ensure there are no further instances of this issue.
We appreciate your patience and apologies for any inconvenience caused.
10:08 - 24th November - Update - We have also identified a fix for issues users are experiencing with logging emails through the Outlook Add-In.
Due to the add-in being hosted by Microsoft, we have had to submit this to Microsoft to authorise the change request. Microsoft haven’t given us a timescale yet on the authorisation, but we’ll keep you up to date as soon as we have more information.
All users should still be able to utilise the Email Logging feature on Outlook within your browser which can be accessed via Office.com
17:26 - 23rd November - Update - Our team have now deployed a fix for the issue with users logging into the Outlook Add-In.
We have also created a fix for the Email Logging and deployed this, however this does require approval from Microsoft for this to be placed live. We have submitted this to Microsoft and aware awaiting their confirmation.
15:06 - 23rd November - Update - We have identified a resolution for the issue with Logging into the Outlook Add In. We have also identified a further issue with email logging however we have also identified a resolution for this.
Our team is currently placing these fixes out to test and will deploy the changes to all users as soon as this has successfully passed our testing.
13:08 - 23rd November - Update - Our team are currently continuing the investigation and have made progress on identifying the root cause of this issue. Work will continue on this as we will post further updates as soon as more information is available.
If you are experiencing issues logging into the Add In within the Outlook Application, follow the below steps to log in within your Internet Browser which should in turn log you in on the App.
If you log in on office.com, you'll be able to access the add-in within Outlook on there and log in - if you close and reopen the Outlook App they should be logged into the Add-In.
11:03 - 23rd November - Update - Our team is continuing the investigation into the root cause of this issue today. We have narrowed down the issue and are working to find a full resolution.
16:03 - Update - We are continuing work to identify the root cause of the Login Issue and as soon as this has been confirmed we will aim to release a fix.
14:00 - Workaround - Users should be able to log in on Outlook on your browser - this should allow you to access the Add-in through office.com.
If you close your Outlook Desktop Application, you can log in to the Add-In within Outlook on your internet browser - this should also log you in on the Outlook App once you reopen it.
12:26 - Update - We have Identified the issue within our Add In and are working to identify the root cause of this issue to resolve.
10.25 - Update- If you are experiencing issues accessing the Add-In within the Outlook application, you will be able to utilise this o Outlook Online. You can access Outlook online by logging into Office.com
10.04 - Investigation - We are currently aware of an issue some users may experience logging into the Outlook Add In. Our technical teams are currently investigating and aiming to resolve ASAP.
30th September - 1st October - Performance Issues
Friday 1st October
1.30am - Update - The re-deployment of infrastructure was completed on the web machines with Windows updates run. All sites checked and confrimed up and running.
We will continue to monitor sites.
Thursday 30th September
10.15pm - Maintenance Release - the maintenance release was completed by 10.15pm this evening, continued monitoring is showing stability across all sites.
3.30pm - Update - Sites continue to remain stable.
Reason: We have invested in upgrading our infrastructure, to increase our web capacity, with the aim of improving the overall speed and experience for all Firefish users and support additional growth. We deployed this new infrastructure on Tuesday evening. All windows updates and checks were run on the new infrastructure before, during and after deployment, and no issues flagged up during any of these checks.
It appears, however, that something in the new infrastructure is defective and is causing a spike in the CPU which in turn has intermittently affected the operation of all Firefish sites. We have been working with Microsoft over the past couple of days, and Microsoft themselves are unable to tell us why these issues are occurring.
Next Steps: We are extending this evenings maintenance release, where we will redeploy a second infrastructure and move all clients over to this, with the expectation that this should resolve the performance issues as the sites will no longer be sitting on the defective Microsoft Azure infrastructure.
We will then continue to monitor this across the night and tomorrow.
1.30pm - Update - Monitoring shows that all sites are currently stable and operating as expected. We will continue to monitor and investigate the root cause.
12.25pm - Update - We have restarted the web boxes and all sites now appear to be operational although slower speeds may still occur for a short time period. We are continuing to monitor the situation and will provide further updates as soon as possible.
12:10pm - Update - Virtual Machines are being restarted. Sites will not be operational whilst the restart is underway.
11:40 - Investigation - We are currently aware of an issue within Firefish which is affecting one of our web boxes. Some Firefish users may experience the system being slower than normal or be unable to access Firefish at this time.
29th September - Performance Issues
13.05pm - Update - Virtual Machines are now fully restarted and all sites are back online. We are continuing to monitor all machines and sites.
12.45pm - Update - We are experiencing further issues when restarting the machines, and have raised a support ticket with our provider Microsoft Azure. It may take a little longer than 1 hour for all sites to be back up and running.
Further updates to follow, thank you for your continued patience.
12.15pm - Update - The Virtual Machines will all need to be restarted one by one, we are in the progress of doing this now and it will take up to an hour to get all sites back up and running.
2pm - Further Reports - We are still seeing issues on one of the Virtual Machines, we are doing further restarts which may lead to slower speeds for some clients. We continue to investigate.
11.30am - Resolved - 17 sites were affected and all resolved.
10.45am - Further Reports - We have had 2 further reports from clients unable to access their sites. We are looking into this now, but it seems to be affecting a very small portion of clients at this time.
10am - Resolution - The virtual machines have now been restarted and we have reports from users that the earlier connectivity issues appear to be resolved. You should now have access to your Firefish site.
We will continue to monitor this situation and provide any further information required. Root cause analysis to follow.
9.50am - Update - The virtual machine has restarted and we now need to restart the second machine to relieve some pressure. Connectivity remains intermittent at the moment and some clients may still experience slower speeds.
9.30am - Update - Initial investigation shows that one of the servers is experiencing issues. We have restarted the virtual machine to relieve pressure.
9.15am - Investigation - We are currently investigating reports that some Firefish sites are unavailable to some users, with others experiencing slower than normal speeds. Our product team are investigating this at the moment and we will keep you updated with findings.
2nd September 2021 - Sites Down
09:55 - Update - The Firefish SQL boxes are now operating as expected and web boxes coming back online. We will now be ensuring that all web sites are up and running but we expect to see continued performance issues for next short while.
Once all sites are confirmed to be back up we will ensure the DB Integrity to make sure that there are no data issues and will also restart all services to ensure that they are operating as expected too.
09:50 - Update - Sites are slowly starting to come back online. If your site is now available, you may experience slow speeds for a short period of time. We will keep monitoring the situation and provide further updates once received.
09:38 - Update - Azure have identified the cause. The Azure service currently remains unavailable and we will continue to update as we receive further information.
Azure Root Cause Analysis - This was caused by an Azure initiated Virtual Machine shutdown triggered by detection of temporary IO transaction timeouts between the physical host node where your VM was running, and the Azure Storage services where your Virtual Hard Disks (VHDs) reside.
Azure platform continuously monitors reads and writes (IO transactions) from your VMs to Azure Storage. If transactions do not complete successfully within 120 seconds (inclusive of retries), the connectivity is considered to be lost and a temporary VM shutdown is initiated to preserve data integrity and prevent corruption of your VM. After the platform detects that the storage service connectivity is restored, the VM is automatically restarted.
09:15 - Update - Azure have an ongoing issue starting at 9:06am where hard disks for machines are unavailable. They are currently working to reestablish connectivity.
This is a region wide issue affecting all Microsoft Azure clients. At present there does not appear to be anything that we are able to do to get any services back on line until we have more information and/or the issue is resolved at Microsoft's end. We will continue to monitor and update periodically with further news.
09.07 - Investigation - We are aware of an issue where it appears that all Firefish sites are currently down. Early indicators suggest this is a problem with Microsoft Azure. We will keep you updated as the investigation progresses.
9th July 2021 - Site Accessibility
11:02 - Resolved - All databases have now been updated and users should experience no further issues.
10:33 - Monitoring - We have now moved all clients to one web box and all accessibility issues should now be resolved. You may still experience minor performance issues and we will continue to monitor.
10:24 - Update - Our team have identified an issue on one of our web boxes which is causing some users to be unable to access Firefish. As part of the resolution, we will be placing all sites on one web box for a brief period. This may result in some minor performance issues for some users.
9:53 - Investigation - We are currently aware of an issue that may cause a small number of users to be unable to access their Firefish database. Our team are currently investigating the cause of this and we will update and resolve it ASAP.
9th July 2021 - Contract Workflow
12:29 Resolved - This resolution has now been deployed to all clients.
11:38 - Update - Testing of this resolution has been completed and we will be working to deploy ASAP.
10:29 - Update - We have identified a resolution for this issue and this is currently out to test. Once passed, we will release it for all users.
09:43 - Investigation - We are aware of an issue making placements within the contract workflow. Our development team are currently investigating the cause of this and will aim to resolve it ASAP.
25th May 2021 - Support Access
11:23 - A Fix has now been implemented for this issue and we can confirm we now have access to the support desk again.
10:50 - We are currently unable to access support tickets due to an issue with the Third Party platform we utilize for our support. The third party are investigation this - see their status page here. We can still be contacted on email@example.com
21st May 2021 - System Speed
11:31 AM - We can confirm that issues some users may have experienced this morning with system speed have now been fully resolved.
You should not experience any further issues as a result of this, we will continue to investigate this internally to avoid a recurrence of this issue.
Thank you for your patience and apologies for any inconvenience.
10:40 AM - We are aware that after some issues last night with Microsoft Azure that some sites may have been running slower than usual intermittently this morning. We are now performing a restart to flush the system. You may experience a period of slowness while we do this which should last no longer than 15 minutes.
10:00 AM - We are aware of an issue that may affect system speed. Our team are urgently working on a resolution and expect to deploy this soon.
20th May 2021 - Site Down
Resolved - 9:08 PM - Microsoft has confirmed all systems are operational.
Monitoring - 7:25 PM - Microsoft are now reporting that most users should now see recovery and they are working to fully resolve.
Update - 6:10 PM - Microsoft Azure is reporting intermittent problems across Europe for key services which are utilized. Unfortunately we are unable to mitigate for these issues. We are reviewing updates from Microsoft as they become available and will update this page throughout.
Update - 5:31 PM - We have investigated this and believe this issue has been caused by an Issue with Microsoft Azure. They are currently experiencing issues with their service which may have an impact on Firefish sites. You can find the Azure status page here. We will update on this as soon as we have further information.
Investigation - 5:15 PM - We have received multiple reports of sites being down and we are currently investigating this.
30th April 2021 - Candidate Record Issue
12:30 PM -We can confirm all issues experienced by users this morning have now been resolved.
We have deployed a resolution and this was completed at 12:25.
We will be working to review the root cause of this issue to ensure we have no further occurrences of this. Apologies for any inconvenience caused.
11:35 - We have successfully tested a resolution to this issue and work has begun to deploy to all users. We expect this to be completed within 1 hour and will confirm when this has been fully resolved. You may experience a slight slow down whilst this is deployed.
9:00AM - We are currently investigating an urgent priority issue affecting candidate records. At present, action history and contact numbers cannot be viewed, and documents are currently unable to be previewed. Our technical team are working on this at the moment.
Please note that a quick workaround which should resolve this issue is to hit CTRL + F5 to refresh the page.
23rd April 2021 - Site Down
Root Cause Analysis: Whilst carrying out a standard procedure, we identified an issue with the SSL certificates against all URL's containing firefishsoftware.com. Once identified, our technical team deployed a fix and all sites were fully restored within 2 hours.
1:06PM - We can confirm this issue has been fully resolved and all systems have been restored and are operating as expected.
11:43AM - All sites are now online. Whilst we work to resolve the overall issue, you may notice some instances of slow speed. We will confirm when this has been fully resolved.
11:35AM - We are working to resolve this for all individual sites affecting. Our technical teams are currently working through all sites to make them live. We will confirm when completed and investigate the root cause of this issue.
11:25AM - We are aware of an issue with some sites being unavailable this morning. Our team are currently investigating this issue and we will update as soon as we have further information. NB this issue is not affecting all clients
27th February - SMS Errors (Resolved)
1.25pm - Twilio (the 3rd Part Integration that supports our Call and SMS features) began experiencing failures with a wide range of their connections.
They began to implement resolutions from 3.15pm and as of 3.30pm most of their services were restored. Voice calling was not affected by these issues. You can find full details of the incident here:
4th September - System Speed Resolution
Further to the speed/access issues, some users experienced yesterday, we have had confirmation that the issue resulted to a localised internet connectivity issue at one of Azure's West Europe Data Centres.
All servers are now operating as expected with both web boxes back to being fully operational. Azures root-cause and mitigation statement can be found here: https://status.azure.com/en-us/status/history/
3rd September - System Speed
3pm - Microsoft Azure has advised they are investigating an intermittent latency or connectivity issue with resources hosted in West Europe, which is likely the root cause of the issues experienced today.
11.08 am - All sites affected by the web box becoming unresponsive are now fully operational as we have moved all sites to the working web box. We will continue investigating the incident to get to the root cause.
10.55 am - Initial investigation appears to show that one of the web boxes has become unresponsive, which is causing some client sites not to load or to cause slower speeds for some users. We have restarted the box within the Azure Portal though it may take some time to restart. We will continue to investigate why this happened once the box has restarted.
10.50 am - We are currently investigating an issue where some sites are offline. We will keep you updated as the investigation progresses.
31st July - Incident Report
During the out of hours weekly release on Thursday 30th July we experienced an outage across a number of client sites.
At approximately 8pm, an unnoticed manual error in the deployment occurred which resulted in the automated weekly release only partially deploying code, resulting in a server error. This error can happen momentarily during the release process, so was not initially picked up as being an issue.
However, around 9pm, we identified that the sites were not recovering as expected and the team commenced an investigation into this. By 9.50pm, we had identified the manual error as the root cause of the problem and tested a fix. This was immediately deployed and all sites were confirmed up and running by 10.15pm.
This morning, we have received our internal documented process and updated to ensure this manual error will not occur again. The team involved in the deployment have also been updated this morning to ensure their full understanding.
17th December 2020
We’re in the process of implementing a new tool that will host all of your help articles as well as make it easier to interact with our team. This is going to be released alongside this evening’s hotfix. Articles will be unavailable from 5pm until around 10pm this evening when the hotfix is released.