Large infrastructure projects are subject to the requirements of a large variety of stakeholders. Requirements can originate from the contract, laws and regulations, operational concepts, site conditions, industrial codes and standards,  the public interest, and other sources [1]. Requirements, which present the desired features and outcomes of a system, can profoundly impact project success as the management of requirements is infamous for being a major source of project failure and going over budgets.

“Projects often suffer several types of problems caused by inadequately managed requirements. For example, contractual and regulatory commitments are overlooked. Requirements do not flow down to subcontractors or suppliers. Requirements are incomplete, unclear, not regularly updated, and subject to widely varying interpretations. ” INCOSE – Managing requirements for Design

The development process for infrastructure projects begins with analyzing and interpreting the requirements of clients and stakeholders, which form the foundation for the integrated facility, system, and components [1]. Contractors rely on the quality of these requirements. One important stakeholder for many contractors in the Netherlands is Rijkswaterstaat (RWS), the Dutch Road Authority. RWS has made significant efforts to capture high-quality requirements by organizing each requirement for civil construction design in separate tables, as shown in Figure 1, rather than in lengthy paragraphs that often require interpretation. Additionally, a unique code has been added to identify each requirement and indicate its origin, applicability to a specific subject or system, and the parent-child relationship between requirements.

This approach has significantly improved the understanding of requirements by clearly separating and identifying them, making them more accessible for reference by specifying the document and the specific code. How a requirement has been derived is now more easily understood through the parent-child relation without the need for interpretation from lengthy paragraphs of text. 

Figure 1: Requirment tables from ”Bouwrichtlijnen Infrastructuur Kunstwerken [2]”

Ongoing challenge

The understanding of requirements is, however, only a component of the larger requirement management process. The requirements of RWS and many other requirement sources like laws, regulations, and standards must be used on every project within the applicable domain. Currently, these requirements have to be taken out of their original source (often textual documents) and put into a project software environment to maintain the verifiability and traceability of requirements over the life cycle of a system. This is a manual effort for every project where the requirement sources are document based. The availability and reusability of the requirements are low due to the manual repetitive effort involved. 

Adopting a machine-readable format for requirements over the document-based form would enable the seamless transfer of requirements to various software systems and the capability to capture additional information, such as verification status, thereby facilitating effective management and control of the system. By implementing this approach, organizations can ensure accuracy, consistency, and efficiency in managing their requirements.

So, despite the effort to make requirements more explicit, requirements are often still recorded in textual documents, requiring manual extraction and transfer to project management systems for each project. This manual process is both time-consuming and repetitive and can lead to inaccuracies.

Taking a leap

Transforming the requirements once to a machine-readable format makes them available on every project without any additional manual effort, increasing reusability. Our solution enables the automatic transformation of requirements captured in a table format within PDF documents into the machine-readable format of Linked Data.

The PDF tables are visually rather than formally structured, making it difficult for machines to interpret the information. Humans have the ability to comprehend and draw connections between the information presented in tables within a PDF file. They can identify similarities in cells and understand that one cell may explain or provide context for another based on location, typeface, or text inside cells. However, machines struggle with this task and require explicit instruction on how to interpret the information captured in tables. This understanding allows us to provide clear and concise instructions to machines, enabling them to interpret the information and support effective requirement management accurately.

There are various ways to tell a machine how to recognize information and classify information captured in a table. There are two main types of information captured in tables, implicit and explicit. When the information is implicit, this means that it is not being mentioned in the table what type of information it is, like a code or requirement text. For example, see Figure 1, in the top row of the table, there are three cells where a value is given but not what kind of information this value is. Contrary to the second, third, and fourth rows, here, the value in the left column defines the kind of information we find in the right column. So for the top row, we have to explicitly tell the machine how to classify the values it finds in each of the cells, but for the lower rows, we only have to help the machine understand that there is a relation between the left and the right column.

Application

Our approach to extracting information from tables focuses on two key functionalities. The first functionality is to reduce the manual and repetitive effort of copy-pasting the information captured in the tables to a requirement system. Our system requires only a small amount of input to indicate which information needs to be extracted and how this information can be recognized. Once these rules are defined, our system can quickly and accurately extract information from complete documents spanning hundreds of pages to a machine-readable format, often in a matter of seconds.

The second functionality is to enable humans to verify and correct the output of the machine easily. Given the critical importance of requirement management, it is essential that the output of the machine be carefully verified to ensure accuracy. To support this objective, our approach displays the extracted data and links the extracted requirement to the table’s location in the PDF document. This allows users to scroll through the document and verify that all the table contents have been extracted. Users can view the extracted data by clicking on a table in the PDF document. By visually presenting the machine’s output, it is simple and intuitive to determine whether all the tables have been extracted correctly.

Figure 2: Extracted table highlighted in PDF document on the right, extracted information in a model on the right.

Figure 3: Requirement hierarchy on the right, requirement text, and link to the PDF document on the right side

In conclusion

Are you aware of the challenges involved in manually extracting well-structured requirements? And looking for a way to improve your process with minimal effort? We have a solution for you. By reaching out to Herman Hoekman, you can automate the extraction of requirements and manage them in a machine-readable format. This approach will enable traceability and provide a semantic-rich way to capture more data.

[1] INCOSE – Managing requirements for Design, https://www.incose.org/docs/default-source/TWG-Documents/003-requirements-management-and-se-pamphlet.pdf?sfvrsn=24c882c6_2

[2] Rijkswaterstaat – Bouwrichtlijnen Infrastructuur Kunstwerken, https://www.rijkswaterstaat.nl/zakelijk/werken-aan-infrastructuur/bouwrichtlijnen-infrastructuur/kunstwerken