logo: Ytria - Essential Tools for IBM Notes and Domino (Lotus)
Ytria Tech Lab
Articles, Tips, and Code for IBM Notes and Domino Administrators & Developers

Un-dead Lotus Notes Documents: How ‘ghosts’ can haunt your databases

on 2/10/10

Have you ever seen a deleted document mysteriously re-appear in a Lotus Notes database? If you answered “yes” you probably weren’t hallucinating … and you’re not alone.

 

"B-b-b-but I thought you were dead! I deleted you months ago!"

 

This eerie issue can occur when someone replicates a database after a long period of dormancy—like when somebody’s laptop gets plugged in after a long vacation. These pesky unwanted docs are commonly called ‘ghosts’ or ‘resurrections.’So what prompts these ghost documents to haunt our Lotus Notes databases?

The short answer: It happens when deletion stubs for deleted documents get ‘purged’ before all replica databases can see them. This means copies of documents that should have been deleted will still remain on a replica database. So during the next replication these documents will be regenerated on the replica databases where they had previously been deleted. But that’s surely not enough to satisfy an inquiring Notes mind, so here’s some more in-depth background information on how ghost documents are generated:

Deletion Stubs in a Nutshell

Every time a document is deleted in Lotus Notes, a deletion stub is created. The deletion stub is used during replication to tell replicas that a particular document was deleted elsewhere, and therefore should be deleted here to keep things in sync.

There isn’t much to a deletion stub.  A stub contains no fields and only the most basic document information (the NoteID, UNID, the Modification Sequence number and the Deletion/Creation dates) necessary to perform its job. But they still do take up disk space so Lotus Notes and Domino offers a provision for deleting—or purging—them after a ‘purge interval’ at which time they’ve (hopefully) outlived their usefulness.

The way Notes handles deletion stubs normally works splendidly…but when troubles arise—watch out!

The Weird World of Purge Intervals

The purge interval is equal to the date set in the Replication>Options>Space Savers (see the image below) dialog in your Lotus Notes client. Oddly enough, it doesn’t matter if you turn on the ‘Remove documents not modified in last __ (days)’ feature—the date here is going to be used as your database’s deletion stub purge interval no matter what. The default purge interval is 90 days.

The weird world of Purge Intervals in Lotus Notes

Purging in Action

Lotus Notes will attempt to purge documents every time 1/3rd of the purge interval elapses. It’s a tad confusing, but think about it this way: If you’re using the default purge interval setting of 90 days, any ‘expired’ stubs lying around in your databases should (ideally) get the heave-ho within 30 days.

Pulling the trigger

Different types of databases have different ‘triggers’ that tell Notes that it’s time to purge expired stubs. Notes checks local databases for expired stubs each time you double-click their Workspace icons. For databases on servers, stubs are purged during the Updall task. Notes handles purging stubs in address books, logs and stats databases a little differently, but we won’t get into that here.
Note: Remotely accessing a database won’t trigger the purge process. Nor will purging be triggered for databases that are already opened by another person or process.

Ghosts in the Machine—Document Resurrection Explained

Under normal circumstances, the way Lotus Notes handles deleted documents and deletion stubs in a distributed environment works splendidly. But when troubles arise—watch out!

Imagine a guy at your organization (let’s call him ‘Bob the HR guy’) takes his laptop with him on a extended vacation/sabbatical.

Upon returning to work, he plugs his computer back into the network and replicates all his local Notes databases. That’s where our troubles begin… There are a few hundred documents on this laptop’s local replicas that had been deleted long, long ago on all the other replica databases (but after Bob went on his sabbatical). The deletion stubs on the other replicas had already expired and had been purged, so as far as Notes is concerned all those old documents on Bob’s laptop are perfectly OK and should be replicated.

Soon people start to notice hundreds of documents that they deleted ages ago are back in the database—like ghosts. (Aside: Though these sorts of documents are commonly called ‘ghosts’ in the Lotus Notes world, I’d argue that calling them ‘zombies’ would be more apt).

Soft deletion: Purgatory for documents

In cases where documents are ‘soft deleted’ (like in the mail database), Notes treats things a little differently. With a soft deletion you get a special ‘limited-time-only’ type of deletion stub that still retains all document data. From this ‘soft deletion’ state the document can be revived and transformed back into a normal document if edited or saved. Once a soft deleted document passes the expiry date (you can set the soft deleted document expiry date in the ‘Advanced’ tab of the Notes application properties dialog), it will be stripped of its field data and transformed into a normal deletion stub. The ‘Empty Trash’ action will also transform a soft deletion into a normal deletion stub.

Ytria scanEZ’s Post Replication Auditor: A Tool for Finding Ghosts

If you have Ytria scanEZ you can use its Post Replication Auditor feature (Tools menu>Post Replication Auditor) to help track down ghost documents in Lotus Notes databases.

The Post Replication Auditor toolbar icon in scanEZ (you can also find it in the Tools menu)

The Post Replication Auditor toolbar icon in scanEZ (you can also find it in the Tools menu)

The Post Replication Auditor is a snap to use, you just need to open the database you want to check for ghosts then click the Audit button. The audit results are shown in a grid that supports grouping and sorting.
Under the hood, the Post Replication Auditor works by analyzing and comparing creation date information for documents that have been replicated and squaring this information with deletion stub lifetime setting (purge interval).

 The Post Replication Auditor dialog. After performing an audit, scanEZ’s Post Replication Auditor will flag documents that are possibly ghosts.

Where did this document come from?

The following four diagrams show four different situations where a new document can appear in a database after replication. In the fourth case, there’s a possibility that the new document is a ghost, so scanEZ will flag it.

|Case #1|

In instances where the ‘Created initially’ date of a replicated document falls within the database’s deletion stub lifetime, the replicated document will not be a ghost.

The document was created within the database's deletion stub lifetime setting (purge interval), therefore it’s definitely not a ghost.

The document was created within the database’s deletion stub lifetime setting (purge interval), therefore it’s definitely not a ghost.

| Case #2 |

When the ‘Created initially’ date matches the ‘Created in this file’ date, it tells us that the document was created in this database (ie the database which you are auditing) so it will not be a ghost.

The document's ‘created initially’ and ‘created in this file' date match, so we can deduce that the document was created in the audited database itself. That means it can’t be a ghost.

The document’s ‘created initially’ and ‘created in this file’ date match, so we can deduce that the document was created in the audited database itself. That means it can’t be a ghost.

| Case #3 |

In instances where the ‘Created initially’ date of a replicated document predates the deletion stub lifetime cutoff date (eg: if the deletion stub lifetime setting was 90 days and this document was ‘created initially’ 95 days ago) but the replica database’s creation date falls within the deletion stub lifetime, the replicated document will not be a ghost.

Here the gap between the ‘created in this file’ and ‘created initially’ dates for the new document exceed the deletion stub lifetime setting/purge interval. But we can still ascertain that the document is not a ghost/resurrection because the replica database’s creation date falls within the purge interval.

Here the gap between the ‘created in this file’ and ‘created initially’ dates for the new document exceed the deletion stub lifetime setting/purge interval. But we can still ascertain that the document is not a ghost/resurrection because the replica database’s creation date falls within the purge interval.

| Case #4 |

In instances where the ‘Created initially’ date of a replicated document predates the deletion stub lifetime cutoff date (eg: if the deletion stub lifetime setting was 90 days and this document was ‘created initially’ 95 days ago) the replicated document could possibly be a ghost.
But remember it’s only possibly a ghost. There are specific instances where one could legitimately expect a document or a design’s ‘Created initially’ date to predate the deletion stub lifetime cutoff date (eg: restored backup documents or a design refresh from a template). We use this icon to indicate a possible ghost/resurrection:This little guy is the icon used to flag Lotus Notes ghost documents/designs in scanEZ

This is where scanEZ will flag a document as a possible resurrection.

This is where scanEZ will flag a document as a possible resurrection.

Warning: Be extra cautious about design elements flagged as ‘Possible Resurrections.’ If you replace or refresh a design, you might see legitimate designs flagged as possible ghosts because their ‘created initially’ dates are old enough to place them outside your database’s deletion stub lifetime setting.

Further reading:

IBM TechNote (Ref# 1091379) – How does purging of documents occur in Notes?
http://www-01.ibm.com/support/docview.wss?uid=swg21091379
IBM TechNote (Ref# 1110117) – Q&As about replication purge intervals and cutoff dates
http://www-01.ibm.com/support/docview.wss?uid=swg21110117

10 Comments

  1. I think calling these reappearing documents “zombies” is better than calling them “ghosts”. Lotus Notes does have ghosts, but those are a different set of documents: the parents of orphaned response documents.
    See http://www-01.ibm.com/support/docview.wss?uid=swg21201427 and http://www-01.ibm.com/support/docview.wss?uid=swg21233933

  2. Ok – so I understand the concept of Ghosts, and we regularly use Ytria to find them, but what do we do when we find a database that really does need documents that fall within the ghost “rules” but we dont want to purge them? We had an example recently where we had to restore 4000 documents. YTria ScanEZ now sees them as ghosts, but they arent…they are relevant restored documents. This now renders the ghost tool less useful because it now wont be able to distinguish real ghosts against our restored documents. Anyone any ideas hwo we resolve without losing the documents original “Created Date”?

  3. @ Marleen
    I agree 100%

    @RobW
    This is a bit of tricky one because, when you think about it, a restored document has every appearance of a ghost/zombie/resurrected doc…the only difference is that we intended for the restored documents to be there.

    There is no silver bullet solution for these situations (that’s why we flag them as “*possible* resurrections” in the grid), but we do have an idea that might help you out:

    Assuming that the restored documents were all created on the same day, you could:
    1) Click the ‘Show Filters’ button in the Post Replication auditor…
    2) Click the filter icon in the ‘Created (in this file)’ column header and select ‘Filter by Regular Expression’
    3) Now enter an appropriate regular expression to filter out the documents created (in this file) on the date of your backup restore; I’d recommend trying a negative “look-around assertion”.
    The specific regex used will vary based on the date format you are using.
    For example, I’m using this type of date format 18/01/2005 12:08:13 PM and the following regex worked for me:

    (?<!08/09/2009)...:..:.....$

    This regex also worked:

    (?<!08/09/2009) .* .*$

    We have a help page on regular expressions here (it also includes links to more advanced regex tutorials):
    http://www.ytria.com/WebSiteHelp.nsf/index/global_RegEx?OpenDocument

  4. …Just to finish, the idea behind that regex is to remove the restored documents from scanEZ’s Post Replication Auditor grid, hopefully making it easier for you to ascertain if the remaining documents are truly ghosts/zombies/resurrections.

  5. Thanks for your response Peter. The RegEx solution is useful but unfortunately not very practical for us. It helps us when we know that there is a date to exclude, but what if we rerun a “ghost” search again in several months time? Those documents will get flagged again and it would be very easy to delete them again. It’s a shame there wasnt a restore flag or date on Notes that could be used somehow.

    Anyway, we have overcome the issue because we were lucky enough to have the document creation date stored in a field on the form, along with a computed document ID that didnt need the system generated Initially Creation Date or UNID. This allowed us to copy the documents and paste them as new documents in our database. Then the Initially Created date falls within the valid rules and doesnt count as a ghost doc anymore. Phew.

    How many days are spent correcting replication problems!!!??? I need BladeRunner to come in and retire all replicants!

  6. Hi Rob,
    So glad to hear you were able to resolve the issue!
    In case you ever need to use the regex solution in conjunction with the Post Replication Auditor the future: scanEZ lets you save regular expressions. So at least you wouldn’t have to re-type them every time.
    Another thing you could do would be to enter the date as a simple “…does not contain” Text filter (available in the same menu as the Regular Expression filter). This approach has the advantage of being quick and very easy, but the the disadvantage is that you cannot save filters for future use the way you can for regular expressions.
    Best,
    Peter

  7. Hi,

    I came here following a google link. I accidentally deleted some design elements I had created in my local copy. It happened when I did a design refresh from a template residing on a different server. There was no replica of my local copy and there is no backup. Do you know any way to get it back? Can you point me to any place where you think I might find solution? Or is is impossible? Thanks.

  8. Hi there,
    I’m sorry to hear about the bind that you’re in. Unfortunately, if you have neither a copy of the database, nor a replica, nor a template, then I believe you’re out of luck.
    Best,
    Peter

  9. Um… aren’t ghost docs (officially) the docs that get created to represent parent docs that have been deleted, so that if you do a “Set parentdoc = db.GetDocumentByUNID(doc.ParentDocumentUNID)” the retuned parent doc will be a ‘ghost’ of the deleted so that the orphan has a representation. This is very different to deletion stubs as these ghost will not get purged.
    Yes, the parent doc that has been deleted will have a deletion stub also; as well as the ghost. Check it out with the bit of LS above.

  10. Love this =0P. I have “ghost” folders that keep being created and they are driving me NUTS! You can’t delete the stupid things.

Leave a Comment

To diplay code in your comment, put the code between these brackets: [cc]your code[/cc]. You can also put inline code by using these: [cci]your inline code[/cci]