Conformance & Interoperability Test & Evaluation, Phase 1

 
As customers in the geospatial industry prepare to modernize their systems based on OGC Web Services, they will require proof of products' conformance to OpenGIS® Specifications in order to achieve interoperability. Validating conformance to an OpenGIS® Specification means verifying that a software product has implemented the specification correctly. Validation involves testing the software interface for response and behavior outlined in the specification. The Conformance and Interoperability Test and Evaluation (CITE) Initiative has provided the geospatial industry (customers and vendors) with a methodology and tools for testing products' conformance to OGC Web Services specifications and for determining whether products that conform are interoperable with one another. Very importantly, CITE also produced conformance tests for the OpenGIS Web Map Service Specification (WMS) and the OpenGIS Web Feature Service Specification (WFS). In addition, CITE produced a conformance engine (a platform for developing tests), a GML validator, scripts and guidelines, and a CITE Portal with links to Websites that implement the official OGC reference implementations of WMS and WFS.

'CITE

For more information on the CITE Initiative, please contact Kurt Buehler (kbuehler [at] opengeospatial [dot] org).

Clarifications

 

1). 9/9/2002 - Questions brought up during CTIE Subcommittee Meetings in Noordwijk
  • A Cost Share/In-Kind worksheet template has been added to the CITE RFQ content. It is also included in the Zip file with the complete RFQ package.
  • The Conformance Test & Interoperability Evaluation Subcommittee discussed the CITE RFQ during the Technical Committee meeting in Noordwijk, the Netherlands (9/9/02). The discussion resulted in the need to clarify several items as follows.

    • "How will the development of the test engine be affected by the planning study?"
      The Planning and Feasibility Study is mostly programmatic and not, necessarily, technical in nature. The schedule is designed such that the Planning Study draft and Initial Conformance Engine design are due coincidentally. This will enable technical issues unearthed during the Planning and Feasibility Study to be addressed by the Conformance Engine developers.
    • "Should proposals include discussion about data sets for testing?"
      Well-formed proposals for the Conformance Test Scripts and Guidelines should address the issue of test data. In other words, preliminary recommendations about data sets should be provided.
    • "Will OGC charge Trademark Licensing Fees to people providing Reference Implementations under the CITE Initiative?"
      OGC will waive Trademark Licensing Fees for Reference Implementations that are accepted for use under the CITE initiative that also pass conformance tests once those are adopted by the OGC for official use.
    • "Will travel costs be covered for those responding to tasks requiring software installation?"
      Please include any expected travel costs in your proposal; we expect to negotiate with you on this matter in order to avoid these costs, where possible.

2). 30 September 2002 - Question:
The specification [RFQ] is vague regarding the requirements for testing the WFS.
It also implies that a WFS tester must also be capable of testing a WMS.
This does not seem reasonable to us, although we appreciate that a general test harness can be useful, and our submission would be extensible to testing WMS also.

We refer to the paragraph in Annex B which reads:
"At a minimum, Guidelines and Test Scripts will be developed for the Web Mapping Server (WMS) and Web Feature Server (WFS) implementation specifications. OGC will also investigate the feasibility of implementing an online Geography Markup Language (GML) validation tool, which would be the method for testing conformance of GML documents generated by the WFS."

Specifically then:

Must any WFS test component also be capable of testing a WMS?

Is the testing of a WFS with respect to the validity of all responses (Capabilities, DescribeFeatures, GetFeature etc.) a Tier 1 activity in this RFQ? Please clarify the last sentence in the paragraph above. Note that GML or GML-related content is contained in many WFS responses (e.g. GetCapabilities, DescribeFeature, GetFeature). Validating the GML from a GetFeature request is only one part of the WFS test.

Clarification:
Proposal responses do not have to include the development of test scripts for both WMS and WFS. The response should include a description of how individual test scripts will be executed with a general test engine.

Conformance tests for WMS, WFS and GML are all Tier 1 requirements. The following statement:
"OGC will also investigate the feasibility of implementing an online Geography Markup Language (GML) validation tool, which would be the method for testing conformance of GML documents generated by the WFS" should be revised to read "Proposal responses should indicate how the Geography Markup Language (GML) validation tool developed in this initiative will be the utilized to test conformance of GML documents generated by
the WFS."

3). 1 October 2002 - Question:
It is clear in the RFQ that OGC seeks test tools for WMS and WFS servers, but it is not clear whether test tools for client applications are also sought. Please clarify.

Clarification:
It is not the specific intent of this RFQ to request such tools.

4). 1 October 2002 - Question:
WBS 8.3 requires a performance test. Against what criteria will performance be assessed?

Clarification:
You must be asking about WBS 8.4.3. This requirement establishes that you will provide an individual to participate in performance tests. The criteria are TBD, however, the primary abstract criteria is:

- Does the implementation properly conformance test the interfaces, operations, encodings, and messages that are within the WMS, WFS, GML and other OGC web-based specifications.

OGC will endeavor to develop more detailed, mutually agreeable performance criteria before the testing occurs. NOTE: Performance Testing is not meant to indicate the execution speed of the implementation.

5). 1 October 2002 - Question:
WBS 9.3 allows delivery of the conformance test system to be by remote access to the developer's site. WBS 12.2.2 requires delivery of source code. Please clarify the requirement.

Clarification:
You must be asking about WBS 9.1 and 12.2.2. The mention of source code in 12.2.2 is intended to refer to the test script specifically and not the test engine, necessarily.