# Architecture Document
## Current Archietcure
### Static View
### Static View Explanations
| Module Name | Associated Files | Description |
|------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Configuration UI | Add.ejs, Edit.ejs, Error.ejs, Home.ejs, Index.ejs, ShowDetailConnector.ejs, Success.ejs | This user interface module that will allow users to add and remove adapters. This user interface module that will allow users to configure adapter configuration and its tool settings. |
| Configuration Module | Add.js, Edit.js, HealthCheck.js, Home.js,Index.js, ShowDetailConnector.js, Success.js, User.js, Create_data.js, Make_request.js, Sorter.js | This module is responsible to pass information that user entered on configuration UI to monitor module. |
| Monitor Module | VDMmonitor.js | This module is responsible to send credentials from Configuration UI to Adapters, and once adapter is successful able to connect with the tool server then this module will write these credentials to Credential document in the database. |
| JIRA Adapter Module | /Adapters/jiraserver.py | This is the adapter module used to connect with the issue tracker, JIRA. It fetches data (Issue Assigned, Issue Numbers, Work Assigned, Team Members) from JIRA using credentials from the monitor module. It implements a heartbeat to Monitor Service to validate it is up and running. |
| Bamboo Adapter Module | /Adapters/bamboo.py | This is the adapter module used to connect with the build server, Bamboo. It fetches data (build status, test cases failed, change in build number) from the Bamboo build server. It implements a heartbeat to Monitor Service to validate it is up and running. |
| Data Processing Module | /dataprocessing/builds, /dataprocessing/issues, /dataprocessing/start.py | This is the data processing module to process the data fetched by adapter modules. It periodically processes the data stored in the CouchDB. It extracts the data relevant to the charts in the application UI and stores the processed data in CouchDB documents (each document for each chart). |
| Application UI Module | /vdm/main/templates, /vdm/main/static, /vdm/main/views.py, /vdm/main/urls.py, /vdm/main/models.py, /vdm/main/buildServer, /vdm/main/issueTracker | This is the application UI module. It allows users to add a new project, and presents project data and organization data. |
### Dynamic View
### Dynamic View Explanations
| Module Name | Description |
|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Configuration | The configuration is a process that sends POST request to the VDM monitors for adapters operation. The configuration process contains user interface part that takes user’s input for configuration of an adapter. In addition, the configuration process also communicate with VDM Monitor process which forwards web request to VDM Monitor process (Add, Edit, Retrieve etc. for adapters) |
| VDM Monitor | The VDM monitor is a process that sends GET REST requests to the adapters for heartbeat information and to refresh the data in the database. Additionally, it sends GET requests to the database to update the scheduler data. It sends POST REST requests to the adapters to reschedule jobs, cancel scheduled jobs, and perform health checks. In return, it receives GET requests from the adapters to send heartbeats. |
| Adapters | The adapters send POST REST requests to the VDM Monitor to signal to them that they are alive and available through a heartbeat mechanism. They also send GET REST requests to their corresponding external servers. These are the third party servers of the tools we currently have adapters for. They also send POST request with credentials to verify authentication of the user on the external tool. Finally, the adapters send POST requests to the database to update the data with the information the adapter pulled. |
| VDM Main | The VDM Main process sends GET REST requests to the CouchDB in order to fetch the data processed by data processing process. After the data is returned, the VDM Main process will render the data on the Application UI. |
| Data Processing | The data processing process sends GET REST requests to the CouchDB every one minute to get the data fetched by the adapters. It will process the data stored in the “builds”, “issues” and “sprints” databases and store the results in the “projects_data” and “build_data” databases using the POST REST requests. |
| Scheduler (DB) | The Scheduler database stores the information about an adapter: authentication, project name, host, port, last update tim, and refresh interval (set in the configuration page). |
| Build Data (DB) | The Build Data database stores the processed data of a Bamboo project. |
| Builds (DB) | The Builds database stores the data fetched from the external Bamboo server. The data will be processed by data processing process. |
| Issues (DB) | The Issues database stores the data fetched from the external JIRA server. It stores the information about issues in the JIRA project: issue id, issue status, assignee, resolve date, due date, sprint id, creation date, estimated story points, logged time (hour), priority, and summary. |
| Sprints (DB) | The Sprints database stores the data fetched from the external JIRA server. It stores the information about sprints in the JIRA project: sprint id in the JIRA server, start date and end date. |
| Project Info (DB) | The Project Info database stores the information about a project: project name in the VDM application, project id in the external server,,description (set in the configuration UI) and the attachment (uploaded in the configuration UI). |
| Project Data (DB) | The Project Data database stores the processed data about a JIRA project. Each document represents a chart shown in the dashboard (part of the VDM Main). The id of each document begins with the project id in the external server and contains the chart name. |
## Architecture Evaluation
### Architectural Drivers: Functional Requirements
The table below outlines the VDM’s functional requirements. For each functional requirement, we assess the current architecture’s ability to meet them.
| Functional Requirements | Relevance to Architecture |
|-----------------------------------------------------------------------|---------------------------|
| Configure adapter and its settings | To configure adapters and see data from its tool, the application follows the “Adapter Initialization process”:
1. First, it creates an adapter service for that tool
2. It then authenticates the user credentials and fetches data from that tool’s external server through the provided API
3. It saves this fetched data to the database
4. Once the data is saved to the database, it is processed by a Data Processor and fed back into the database
5. Finally, the data is ready to be rendered for display on the web browser
In the current architecture, each adapter is expected to be a separate module which internally handles steps 1-3 of the process. There is in fact a separate Data Processor process that is able to process the data prior to being rendered for the front-end. This is a sufficient satisfaction of the functional requirement.
**Additional considerations**:
Currently the adapter modules experience low cohesion. The individual adapters are doing too many subtasks, and this could be duplicated across adapters. This puts a risk on maintainability, because we must ensure that if the code is updated in one adapter, that it must reflect in all of them if necessary. We should consider pulling out duplicate, redundant code into a separate module.
| Integrate new adapters | Currently, new adapters can be added only if dependent modules are updated. We can see from both the static and the dynamic views that the following must be updated:
1. VDM monitor should be updated to include the boolean “firstTime” flag.
2. Add a new adapter process. Currently the adapters are written in Python.
**Additional considerations**:
1. Currently, there are flags for only JIRA and Bamboo to determine if this is the first time we are starting up an adapter. Long-term, we should update this so that there is perhaps a map that holds flags for connectors. This would allow us to add adapters without hardcoding new flags every time.
2. The database must support build and issue information that fit JIRA and Bamboo (Atlassian Suite-specific) data patterns and models only. That is to say, if we add a new adapter for different issue trackers, build systems, or a new adapter for code repository altogether, then we may have to make changes to account for them. We should preemptively make sure that the database can support these changes.
3. Long-term, adapter modules responsible for fetching data from external tools should be as loosely coupled as possible with the core components of the application. Currently, there are some dependencies that must be addressed.
| Remove tool configuration | The individual adapters are currently in charge of supporting cancellations of connector instances. The cancellation is executed through the web browser, which is translated to the VDM monitor, which in turn notifies the adapters through GET.
**Additional considerations**:
The considerations are the same as the above two squares. Most importantly, the adapter modules should be as loosely coupled as possible with the core application components.
| Fetch projects and related information | Once the adapter’s configuration is configured from above mentioned “Adapter Initialization process,” the individual adapter will be in charge of fetching project related data. The current design makes updateData call from VDM monitor to adapter file, the updateData call will make a request to python file which will force the backend to grab the up-to-date information from the external server. So, the basic fetching information functionality is achieved. They also provided a documentation including what are the must have APIs in the adapter.
**Additional considerations**:
The current architecture design does not include data validator according to our conformance check the static view. There should be some sanity check before the adapter saves project information into the database.
| Combine data associated with a project | Currently, the project data is pulled from the adapter into the database, and there is a data processing service running in the backend to sort the information in the database. After the information sorted for each project, the user interface side can render the project on screen quicker. We feel like this is sufficient design to meet the functional requirements.
**Additional considerations**:
The current design of the data processing service has some negative impact on the performance of the application. The detailed project information will not be available until
1. Adapter pulled all project and issues/builds from external server 2. Data processing service read from database and process issues/builds for that particular project
If we can combine above two steps together, we can achieve faster backend processing.
| Update data based on refresh interval | Currently, there are two refresh intervals implemented in their application. One is refresh interval on the web page, and the other interval is refresh interval for backend to retrieve new data from the external server.
**Additional considerations**:
Currently, there are two different refresh time interval settings. To be more clear with the user and have more consist data, we should have two intervals combined.
| Delete project data when application is shutdown or the adapter fails | The current design is not deleting the project data either in the situation of failure of the adapter or application shutdown. If adapter fails to connect, the backend service will try to restart adapter instead of deleting project data. The only time that information will be deleted is destroying the vagrant development box or deleting the adapter from configuration page.
**Additional considerations**:
The current design is clearly not matching what is described in the functional requirements. We need to implement the deletion logic in VDM monitor service to ensure the project data gets deleted. |
## Architectural Drivers: Quality Attributes
In addition to functional requirements, there are several relevant quality attributes/non-functional requirements that must be addressed.
### Quality Attributes
* **Performance**: Performance is measured by the system’s response time; we must ensure that a request is fulfilled in a pre-specified amount of time for various scenarios.
* **Reliability**: Reliability refers to VDM’s ability to perform operations and functional requirements consistently. This includes pulling the appropriate data for a given project on the correctly set intervals. In other words, its ability to perform these functions should be reliable.
* **Availability**: If a connector becomes unavailable, then the application should be notified and be presented the option to provision a new connector.
* **Modifiability**: Since one of the goals of this project to build open-source community around this tool, the application should be modifiable. A developers wanting to contribute and integrate a new adapter should be able to do so without modifying the application’s core modules.
* **Interoperability**: The developers should not be limited to NodeJS when choosing a language to write a new adapter for. Further exploration is required to figure out which languages VDM will support.
### Quality Attribute Scenarios
#### QA1. Performance - Time Behavior (Transaction Time)
During normal operation when the application is running without error and failure, when a user clicks on the “refresh” button, the application should be able to render latest data on the browser in a minute. This includes fetching data from the external tools, adapters saving it to the database, then adapter service retrieving it, and finally rendering the data on the webpage. Thus, the request-response transaction time should be under 1 minute.
#### QA2. Performance - Time Behavior (Adapter Initialization)
**Assumption**: Assuming that the application already support that tool and has the adapter for that tool. During normal operation and the application is running without error and failure, when a user connects with a tool for the first time, the application should be able to spawn adapter service for that tool, authenticate credentials, fetch data from the tool, and render it on the browser in 2 minutes.
#### QA3. Performance - Time Behavior (Adapter Addition)
**Assumption**: Assuming that the application already supports that tool and has the adapter for that tool.
During normal operation when the application is running without error and failure, when the application spawns a new adapter service, then there should be no impact on the performance, basically the application be able to service requests within a minute for existing tools, request-response transaction time should be under one minute consistently.
**Justification**: We can avoid performance penalty with the addition of a new adapter by making sure that there is no dependency between contributed adapters and each one can fetch data from the tools concurrently. To minimize dependency, we have a database, which will store fetched data and a data processing service to collate all the data together before sending it to the user interface component.
#### QA4. Reliability
**Assumption**: The tool has been successfully integrated with the application by completing the configuration setting steps.
During normal operation, when an adapter becomes unavailable then the application should:
1. Notify the user in 90 seconds, which includes two missed heartbeats of 30 seconds and the remaining 30 seconds to display the alert message
2. Provision a new adapter (same as failed one using cold spare) within 90 seconds of user notification.
#### QA5. Modifiability
**Assumption**: The developer has already developed a connector to fetch data from the external tool like GitHub, Jenkins, and Jira and this connector will call “Data management layer” to pass the fetched data to VDM’s application layer. There will be detailed instructions provided on VDM’s website that will walk the developer through how to commit their connector to application’s connector repository.
During normal operation, the developer should be able to successfully integrate the newly created adapter with the application without making any modifications to the core modules. The developer should be able to see the data fetched from the newly integrated tool once Adapter Initialization process has been completed.
To integrate a new adapter a developer can modify the application layer, specifically adding the new module to the contributed modules repository, as shown in the static diagram figure 6 below.
#### QA6. Interoperability
**Assumption**: The developer has build a new adapter which uses the APIs provided by the application.
When the application is not running, a user wants to add a new adapter to the application and the adapter is developed in a programming language other than Node.js. The developer should be able to integrate the new adapter to the system such that the adapter becomes available in the configuration page after the user has downloaded and restarted the application.
### Prioritization
| Rank | Quality Attribute | Justification for Priority |
|------|--------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1 | Modifiability | Modifiability directly maps to one of the project’s primary goals to build an open-source tool. It is essential that developers should be able to integrate new adapters with the application with minimal amount of modifications to the application’s code and with ease. |
| 2 | Reliability | The application’s value comes from its ability to consistently fulfill its functional requirements. If the application cannot do so, then there is nothing stopping from users from just using the tools directly. Our application must be reliable to add value. |
| 3 | Availability | If an adapter fails, the system can recover gracefully by providing the user with some option to provision a new adapter, so that system have the data from the required tools. This is related to reliability in that the VDM’s livelihood depends on the value it adds to its users. If the application is constantly unavailable, then there is no reason for users not to use the third party tools directly. |
| 4 | Performance - Transaction Time | The customer should have the latest data from external tools on the application every minute. Since we are currently focused on ensuring the tool is functional, performance has been de-prioritized. |
| 5 | Performance - Adapter Addition | Since the customer wants the application to support multiple adapters, it is important to ensure that any additional service does not impact the required response time of 1 minute of the application. |
| 6 | Performance - Adapter Initialization | When a new adapter are added, it should display the data on the application within 2 minutes. This is the least crucial of the performance quality attributes, because even if initialization takes more than 2 minutes, it will not have any significant impact on the business goals. |
| 7 | Interoperability | Since we will be opening the tool up to the open source community, it would be better to support more languages without restricting development to a single programming language. However, this is relatively lower in priority, since we can always add support for languages later once the core functionality has been implemented, tested, and deployed. |
## Tradeoffs/Technical Debt
* **Data Validator**: Currently we are missing a module to validate data prior to storing it in the database. We have decided to prioritize functional bug fixing over making this architectural change at the request of the client, however this should be done in the near future, because it greatly inhibits the ability for the application to scale. This is also important for when we deploy
* **Module Restrictions**: In the context of modifiability, we may need to employ some mechanism to control modifications to core modules when opening this to the open source community.
* **Ensure Support for Additional Issue Trackers**: Assess more APIs for issue tracking systems other than JIRA.
* **Interoperability Evaluation**: It is currently unclear what languages are supported by the application for building new adapters. We know for sure that Python is supported (this is the language that the Bamboo and JIRA adapters were built with).