Skip Ribbon Commands
Skip to main content

Recovering Workflow History after 60 days

Or finding where it's been hidden...

Andrew Walman

10/10/2011


​​​​Workflow history is one of two things to SharePoint users. I suspect for the majority, they aren't even aware of it, and don't have any need for it. For a small minority, they are fully aware of it for one important reason – compliance. Workflows are nearly always used as part of a larger business process, and often these business processes require auditing to ensure compliance with whatever industry standard the business operates with. When SharePoint workflows are used, workflow history is usually seized upon by users as being an easy way to audit compliance. After all, it's easy to get at isn't it?

Certainly while the workflow is in progress, and for a couple of months afterwards it is. But what happens when the end of year audit arrives, and the auditor goes to look at the history for a workflow completed 11 months ago? Disaster – it's not there! There must have been many a frantic phone call between panicking compliance officers and frustrated IT staff along the lines of this:

  • [Compliance officer] "My workflow history has been deleted! The auditor is going to fail us! Where has it gone?"
  • [IT support] (probably after Googling "Workflow history deleted") "This is by design. SharePoint deletes workflow history after 60 days"
  • [Compliance officer] "WHAT?! Nobody told me that! Can you recover it?"
  • [IT support] "We can, we'll just need turn off the timer job and then restore the entire SharePoint farm for each month in question."
  • [Compliance officer] "How long will that take?"
  • [IT support] "Hmmm…."

Both parties are at fault here. The compliance officer for writing a feature of SharePoint into the compliance process without knowing how it worked, and IT for not explaining the 60 day "deletion" actually happens. The good news is though, workflow history isn't deleted, all we need to do is make it easier to find.

So Where Is Workflow History after 60 days?

Put simply, it's in the same place as it's always been. There's a hidden list in every SharePoint site with the workflow feature activated called "workflow history" and it's under lists, so pointing your browser at http://[sitename]/[subweb]/lists/workflow%20history/ will show you this list. It's not pretty, and certainly not usable by an end user in its default view, but it is the same content that was available from the item that the original workflow ran on. All that happens after a workflow has been closed for 60 days is that the relationship links between the item (list item or document) and the workflow history are removed from the database by a timer job. The reason this is done is that a workflow can consist of many individual steps, each of which gets recorded in the history list. Maintaining those links in the database for every workflow that ever runs in a site slows performance down. So Microsoft implemented a clean-up job to remove the links, making it appear from the item that the workflow history is gone. It's possible to disable the timer job, but this has to be done at the application level, will kill performance in the long term, and doesn't help you get back the "missing" workflow history.

Making Workflow History Usable Again

Knowing that all Workflow History is available in a list, all we need to do is link it back to the original item using a calculated column and a view. The basic steps are:

  • Create a view of the workflow history list that uses the filtered ID to present the history of a particular workflow in a recognisable format.
  • Create a new calculated column in the list or library that is associated with the workflow.
  • Write a formula in the calculated column that inserts the item/document ID into a link that can be passed to a view of the workflow history list.

Once we've completed these steps, any user who can open the item will be able to see the new link to the workflow history and view that. Now for the nitty gritty:

Create views of the Workflow History List

  1. Open the workflow history list at http://appname/subweb/workflow%20history/
  2. Create a new shared view; you'll need one for each list that has a workflow associated with it – if there's only one, call the new view "audit view", otherwise "workflow name – audit view" or something similar.
  3. You'll need to identify the List or Library ID (GUID) – You can use the full history list for this.
  4. As a minimum, add the following columns to the view:

    Date Occurred

    User ID

    Event Type

    Outcome

    Description

  5. Sort the view by "Date Occurred"
  6. Filter the view by "List ID" (List ID is equal to GUID – include the braces)
  7. Group by "Primary Item ID", then by "Workflow History Parent Instance"
  8. Save the view, try it out – add any other customisations your audit process may need.

At this point the view will return all history for a particular list, in the next steps we'll create a link that opens and filters workflow history for a particular item.

Create the calculated column for a list/library

  1. Within the list that is associated with the workflow, add a new calculated column. Do this for each list from which you need to see the workflow history.
  2. Enter the following formula, replacing appname and subweb with the address of your site, and viewname with the view you just created:
    ​​ =CONCATENATE("http://appname/subweb/Lists/Workflow%20History/viewname.aspx?&FilterField1=Item&FilterValue1=",ID)
    ​​​
  3. Add this column to the default view, or create an "auditor's view" of the list, containing this column and any other pertinent information.
  4. Now users can click on the link created dynamically by this column to return a filtered view of the workflow history, containing audit information on each step in every workflow that ran for this list item.

 
​Edit:​​​ Further to the comments below, to make the link clickable, you'll need to edit the formula as follows, using single quotes in the HTML, and ensure the output is set to number​, not text. e.g:
​​ =CONCATENATE("<a href='http://appname/subweb/Lists/Workflow%20History/viewname.aspx?&FilterField1=Item&FilterValue1=",ID,"'>Link To Document</a>")​

Make the workflow history easier to read

Out of the box, the workflow history contains a lot of GUIDs instead of real names (I suppose this is where deleting all those links made sense to Microsoft). To make it more human legible, you can add further calculated columns to the workflow history list to turn some of those column values back into real names. A good example is the workflow name, which is represented by "workflow association ID"

  1. Add a new calculated column called "Workflow name" to the workflow history list.
  2. Use the following formula:
    =IF([Workflow Association ID]="{GUID of particular workflow}", "Name of particular workflow",IF([Workflow Association ID]="{GUID of another particular workflow}", "Name of another particular workflow"))
    ​​
  3. Now save and add this column to your workflow audit views.

That's all. It's clearly not as straightforward as being able to use the OOTB workflow history from the item, but if it means you can still access the same information anytime without harming the performance of the database, it's definitely worth doing! If you know you're going to need workflow history for multiple sites, it's probably worth adding the views at least to the site template before the sites are created, and you could even add the columns to the default libraries with some placeholder values to aid deployment.


Top Blog Posts From Fuse

 

 

Simple Incoming Email with On Premises SharePoint and Exchangehttps://www.fusecollaboration.com/blog/simple-incoming-email-with-on-premises-sharepoint-and-exchangeSimple Incoming Email with On Premises SharePoint and Exchange
Recovering Workflow History after 60 dayshttps://www.fusecollaboration.com/blog/recovering-workflow-history-after-60-daysRecovering Workflow History after 60 days
Allowing anonymous users to add items to SharePoint listshttps://www.fusecollaboration.com/blog/allowing-anonymous-users-to-add-items-to-sharepoint-listsAllowing anonymous users to add items to SharePoint lists
How to use SaaS solutions to identify sensitive datahttps://www.fusecollaboration.com/blog/how-to-use-saas-solutions-to-identify-sensitive-dataHow to use SaaS solutions to identify sensitive data
SharePoint Web Application - Empty Virtual Directoryhttps://www.fusecollaboration.com/blog/sharepoint-web-application-empty-virtual-directorySharePoint Web Application - Empty Virtual Directory

Recommended Pages

 

 

About Fuse Collaborationhttps://www.fusecollaboration.com/fuse-home/aboutAbout Fuse Collaboration
Hosted Skype for Businesshttps://www.fusecollaboration.com/technologies/skype-for-business/hosted-skype-for-businessHosted Skype for Business
Skype for Businesshttps://www.fusecollaboration.com/technologies/skype-for-businessSkype for Business
Microsoft 365 Enterprisehttps://www.fusecollaboration.com/technologies/office-365/microsoft-365-enterpriseMicrosoft 365 Enterprise
Fuse Collaboration SharePoint Consulting, Design and Developmenthttps://www.fusecollaboration.com/technologies/sharepointFuse Collaboration SharePoint Consulting, Design and Development

 About us

Fuse Collaboration Services is a Cloud Solution Provider and Microsoft Gold Partner specialising in delivering SharePoint, Skype for Business, and Azure cloud-based solutions. Based in Northampton, UK.

Microsoft Gold Partner Logo showing 5 competencies

Read more

 Latest Tweets

 Latest Blog

 

 

How to use SaaS solutions to identify sensitive data1497<p class="lead">​​​​​​​​​​​This article is going to look specifically at how we implement the use of software (SaaS) to enable your organisation to become ready for the GDPR quickly and easily, without interruption to your end users.</p><p> <strong>The first step in getting ready for the GDPR is to know what data your organisation holds</strong>. At the time of writing this article, the new legislation is only <strong>268 </strong>days away and the four main questions you n​eed to be able to answer to ensure your organisation is ready are&#58;</p><ul><li>What data does your organisation hold?</li><li>Where is the data kept?</li><li>Why do you need to use or keep the data?</li><li>Do you have consent to use the data?​</li></ul><div class="thumbnail"> <img class="img-responsive" alt="A padlock on a background of binary data" src="/ourblog/PublishingImages/Pages/How-to-use-SaaS-solutions-to-identify-your-data,-ensuring-your-organisation-is-ready-for-the-GDPR/shutterstock%20Data%20protection%20GDPR%20blog.jpg" style="max-width&#58;500px;" />​</div>​ <h3>What is defined as sensitive data under the GDPR?</h3><p>In terms of the GDPR, sensitive data is defined as personal data, but goes further than the Data Protection Act and includes online identifiers such as an IP address. The GDPR applies to both automated personal data and manual filing systems. You can be held responsible for breaching the GDPR by allowing personal data to be compromised either by&#58;</p><p>Misuse - using data for purposes other than that defined and recorded&#160;consent given for;</p><p>or </p><p>for data breaches, even if the breach was a malicious act (hacking), if you can't prove you had adequate data security measures and processes.&#160;​</p><div class="row"><div class="col-md-6"><h4>Will my company have GDPR sensitive data?</h4><p>Data that can identify any individual, such as a name, National Insurance number, passport, IP address or even biometric data - a soon as this is recorded in any system, in a file, a database record, or even on paper - that data then falls under the remit of the GDPR. As every organisation has staff records, every company will be affected by GDPR to some extent. </p><p>However, the more individuals you deal with, and the longer you hold that data for, the more prone you are to breaches of the legislation. Companies that perform data processing, even on behalf of other companies, and particularly those that use personal data records for multiple purposes (for example re-marketing) are at most exposure to GDPR.</p></div> <div class="col-md-6"><h4>​Examples of who will be most affected&#58;&#160;</h4><ul><li>Retailers – High street shops and online retailers storing customer profiles</li><li>Health Sector -&#160; Hospitals, doctor’s surgeries, scientific research organisations, pharmaceutical companies, with patient records</li><li>Education sector - Schools, colleges and universities, storing current and past student records</li><li>Financial sector – Banks, mortgage and insurance providers, with customer accounts</li><li>Recruitment companies - candidate records</li><li>Charity organisations - records of donors and recipients</li><li>Estate Agents - vendor and client records </li><li>Legal profession – Solicitors, CPS and courts - client records.</li></ul> ​</div></div><hr />​ <h3>​How to identify data and ensure all your data is GDPR compliant?&#160;</h3><p>There is an easy and quick way to find out what data you hold and you will be relieved to know we work alongside companies that are currently releasing SaaS solutions that are designed purely to scan, discover and analyse your data, to ensure you only hold data that is GDPR compliant.&#160;&#160;<br></p><p>Our Partners have solutions that use metadata to scan and analyse data which has enabled the migration of data to SharePoint for some time. It’s this technology that’s enabled these new solutions to be <strong>created specifically for the use of identifying what data you have and if it is GDPR compliant</strong>. Using NPL (Natural programming language) such as “name”, “address” or “credit card number”) this process can be done in days not months and can easily identify documents in unstructured databases, file shares and SharePoint. </p><p>The discovery phase of the SaaS tool is an important part of the new solutions as they are designed around common datatypes that can be tagged easily e.g. names, addresses, age of document, author of document, credit card numbers, postcodes IP addresses. The solution we use comes with predetermined taxonomies which can be edited easily, to reflect the sector that your organisation works within. Dashboards are then accessed with detailed data analysis which identifies the data that will not be compliant with the GDPR. Additional columns appear alongside your files with a “true” or “false” label showing whether the files are compliant with the GDPR and our team of consultants are experienced in using this technology and can advise you depending on your specific IT infrastructure.</p><hr /><h3>Hype around the GDPR&#160;</h3><p>There is a lot of hype around the new GDPR coming out in May which appears to be fairly negative but instead of viewing it as a tiresome challenge that your organisation must overcome, I would advise viewing it as an excellent opportunity to gain a competitive edge within the market. Whatever your opinion is and I doubt there are many companies that relish the additional resources that will be needed to comply; from a customer’s perspective, it <strong>must </strong>be a good thing. The trust in any business relationship is one of the fundamental reasons why you have a successful company. In an age where we frequently see headlines describing yet another data breach, damage to a business’s brand and reputation is an expensive result of avoiding being compliant with the new regulations.</p><hr /><h3>What to do right now!&#160;</h3><p>The key decision makers in your organisation need to be made aware of the GDPR. They need to know that the first thing they need to do is to find out what data their organisation holds. This is where we come in.</p><p>Although GDPR isn’t an IT issue as it will ultimately fall to the responsibility of who is currently responsible for your Data Protection E.g. Compliance Managers, Data Protection Officers, Data Controllers and Office Managers, the team at Fuse will be able to provide a technical solution to determining what data is held.</p><p>Before any amendments to existing internal procedures, policies or customer facing documentation such as websites and application forms can be changed they must know what is relevant to the GDPR. It may be the case that a lot of the data you hold is ROT and this can simply be deleted. If you have a completely unstructured filing system and want to take the opportunity to improve the efficiency of your business we don't just offer technical expertise, we are experienced in developing proofs of concept, functional and technical specifications. We can either then take responsibility for delivering the project or work alongside your IT departments providing a technical lead.</p><p>It doesn’t matter whether you have an inhouse IT department or not as we can work alongside existing IT managers, Compliance or HR managers. An IT consultancy needs to be your first port of call as they can advise you as to which is the best SaaS solution for your business depending on the size of your business and your budget. Having expert knowledge of your IT infrastructure and how it works is important to ensure that the right solutions are used. The benefits of using SaaS is that your end users are not interrupted and your IT departments are not impacted either. </p><p>You need to weigh up the cost of using an IT consultancy who can implement the right tools against the extra resources it will take to trawl through and analyse your data manually. This can seem daunting, but it’s a great opportunity to get your data in order and have confidence that your organisation can be proud of its commitment to protecting the data of its employees, customers and suppliers.​</p><div class="well well-lg"><p class="lead">If you want further advice or a quote on how we can help you get ready for the GDPR call Fuse today on 01604 797979 or <a href="/_layouts/15/FIXUPREDIRECT.ASPX?WebId=4fc45909-2b6d-48b9-bcf9-a446e9d472d6&amp;TermSetId=c98895cd-d37f-4406-9cff-5480b4f829b6&amp;TermId=218eb0be-10f6-490a-82a7-a7fd47c8de90">contact us​</a></p></div>​l.ozier@fusecollaboration.com | Louise Ozier | 693A30232E777C6675736563735C6C2E6F7A696572 i:0#.w|fusecs\l.ozier28/08/2017 23:00:002017-08-28T23:00:00Z Ensuring your organisation is ready for the GDPR19/09/2017 23:16:251671htmlFalseaspx

 Contact us

Our address
12-14 Brookfield, Duncan Close
Moulton Park, Northampton
NN3 6WL
P: +44(0)1604 797979
Contact Us